Interactive Explainer
Random Forests, Tree by Tree
Bagging + random feature sampling — the simplest tabular ensemble that still wins half the time. Add trees one at a time; watch the boundary smooth, OOB error fall, feature importance stabilise. Compare RF to bagging-only and Extra-Trees on the same dataset.
The variance-reduction argument in two lines
A single deep tree is high-variance: small training perturbations move splits a lot. If we have $T$ trees with pairwise correlation $\rho$ and per-tree variance $\sigma^2$, the variance of the average prediction is
Two terms: an irreducible part driven by between-tree correlation, and a $1/T$ part driven by independence. Bagging lowers $\rho$ by training each tree on a different bootstrap sample. Random feature subsampling lowers $\rho$ further by forcing different trees to split on different features. The combination is what makes a random forest stable: $T \to \infty$ drives the $\sigma^2/T$ term to zero, and the residual $\rho\sigma^2$ is much smaller than $\sigma^2$.
A single deep tree is unbiased-ish but variance-heavy; averaging hundreds of them keeps the bias and crushes the variance. This is the entire intuition behind every bagged ensemble.
Two sources of diversity
A single deep decision tree is high-variance — small training-set perturbations move the splits a lot. Random Forest (Breiman, 2001) averages many such trees, each de-correlated from the others by two tricks:
- Bagging. Each tree sees a bootstrap sample of the training data (sampled with replacement, same size as the original). About 63% of unique points appear in each bootstrap; the other 37% are the out-of-bag samples for that tree — a free held-out set per tree.
- Random feature subsampling. At each split, only $m$ of the $d$ features are eligible to be picked. Defaults: $m = \sqrt d$ for classification, $m = d/3$ for regression. Forces different trees to look at different features and beats bagging-on-features-alone.
Extra-Trees (Geurts et al., 2006) goes further: at each split, pick the threshold randomly rather than optimally. Trades a little bias for even more variance reduction; often slightly worse on signal-rich data, slightly better on noisy data.
Watch the forest grow
Click Add tree to add one tree at a time. The forest's predicted class is the majority vote of its trees. The boundary smooths quickly in the first 10 trees and gradually after that. OOB error is reported only once $\ge$ 3 trees exist.
OOB error — the free held-out set
Each training point is "out-of-bag" for the ~37% of trees whose bootstrap didn't include it. To get its OOB prediction: vote across just those trees. The OOB error is the misclassification rate over all training points using their respective OOB predictions.
Properties: (1) no leakage; (2) you don't have to split into train/test for tuning; (3) converges to the true test error as trees grow. The OOB curve in Step 2 should match the slope of a held-out test curve within noise.
Feature importance — three ways, none perfect
- Mean decrease in impurity (MDI). Sum up the impurity drop a feature contributed at every split in every tree; normalise. Fast, biased toward high-cardinality features.
- Permutation importance. Shuffle one feature in the OOB set; measure the drop in OOB accuracy; larger drop = more important. Unbiased, expensive.
- SHAP / TreeSHAP. Per-prediction contributions with Shapley-value guarantees. The standard modern way to explain RF / boosting predictions.
MDI is what the live demo shows. Always cross-check with permutation importance on real datasets — MDI can flag a random-noise high-cardinality feature as "important".
What RF is good at — and what trips it up
- Strengths. Mixed-type tabular data, many features, mild interactions, robust to outliers, handles missing values, parallel training, OOB tuning.
- Weaknesses. Extrapolation outside training range (zero by construction); smooth low-dim regression (a GP is better); highly imbalanced classes (use class-weighted sampling); very high-dim sparse (XGBoost / LightGBM tend to win); cannot capture rotated decision boundaries efficiently.
- Where it sits in 2026. Still the default first-try on any tabular task you can throw 10 lines of sklearn at. If you want one more lap of accuracy, switch to XGBoost / LightGBM and tune. If you need probabilistic outputs, wrap with conformal prediction or use NGBoost.
RF vs Gradient Boosting — when to use which
| Axis | Random Forest | Gradient Boosting (XGBoost / LightGBM) |
|---|---|---|
| Training | Trees are independent (parallel) | Trees are sequential (later trees fix earlier mistakes) |
| Per-tree depth | Deep (often unlimited) | Shallow (3–8) |
| Sensitivity to hyperparameters | Robust — defaults usually work | Sensitive — must tune learning rate, max_depth, regulariser |
| Risk of overfit | Low (averaging) | Higher — need early stopping |
| Calibration | Poorly calibrated probabilities | Slightly better, still wrap with Platt / isotonic |
| Best on | Many features, mild signal, fast prototyping | Tight accuracy targets, large datasets, Kaggle |
Mental model: RF averages many trained-from-scratch trees; boosting trains a sequence of small trees on each other's residuals. See the XGBoost article for boosting's mechanics.
Hyperparameter cheat sheet
| Parameter | Default | What to change & when |
|---|---|---|
n_estimators | 100 | Increase until OOB error plateaus; cheap upside, no overfit. |
max_depth | None (full) | Cap at 10–15 if memory matters; small effect on accuracy. |
max_features | $\sqrt d$ (clf) / $d/3$ (reg) | Lower for redundant features; raise for sparse / weak features. |
min_samples_leaf | 1 | Raise to 5 if labels are noisy; smoother boundary. |
bootstrap | True | Set False to disable bagging (then OOB is undefined). |
class_weight | None | Use "balanced" for imbalanced classification. |
n_jobs | 1 | -1 to use all CPU cores — embarrassingly parallel. |
Reading list
- Breiman (2001) — Random Forests. The original paper. Highly readable.
- Geurts, Ernst, Wehenkel (2006) — Extremely Randomized Trees.
- Hastie, Tibshirani, Friedman, The Elements of Statistical Learning, ch. 15.
- Louppe (2014) — Understanding Random Forests: From Theory to Practice. PhD thesis, depth on bias-variance and importance estimators.
- Strobl et al. (2007) — Bias in random forest variable importance measures. The MDI bias paper; canonical reference for "use permutation importance instead".
- Grinsztajn, Oyallon, Varoquaux (2022) — Why do tree-based models still outperform deep learning on tabular data?