Interactive Explainer

XGBoost, Tree by Tree

Gradient boosting in slow motion: each new tree fits the residual of the previous ensemble. Click Add tree; watch the prediction sharpen and the loss drop. Then see why the second-order (Newton) update XGBoost actually uses beats the vanilla first-order one.

~18 min Classical ML · Boosting · Trees

Step 1

The boosting recipe in two lines

Build a sequence of weak learners (small trees), each fitting the residual of the running prediction. The ensemble prediction is the sum of all trees.

$\eta$ is the learning rate. Smaller $\eta$ + more trees is almost always better than larger $\eta$ + fewer trees, up to a budget. $f_t$ is what tree $t$ outputs on input $x$.

What XGBoost adds

Vanilla gradient boosting fits $f_t$ to the negative gradient $-\partial L / \partial \hat y$ (a first-order approximation). XGBoost (Chen & Guestrin, 2016) does a second-order Taylor expansion and fits the tree to optimise the Newton-step direction. Its split-gain and leaf-value formulas are:

where $g_i = \partial L / \partial \hat y_i$ and $h_i = \partial^2 L / \partial \hat y_i^2$. For squared loss $L = \frac{1}{2}(y - \hat y)^2$ this reduces to vanilla GBM ($g = \hat y - y$, $h = 1$). For logistic loss XGBoost's Hessian-aware update converges meaningfully faster.

Step 2

Tree by tree, live

A 1-D regression toy with a hidden curve. Click Add tree; the next tree fits the current residual; the ensemble prediction (teal) updates. Compare to a single deep tree and a random forest of 25 trees on the same data.

learning rate η 0.30 max depth 3 trees 0 train RMSE —

Training data (●), true curve (dashed), boosted ensemble (teal solid), latest tree's contribution (orange dashed).

Residuals after the current ensemble. Boosting drives this toward zero.

Train RMSE vs number of trees. Watch the elbow shape.

Compare to single tree + random forest

single deep tree (depth 8)

random forest (25 trees, depth 6)

boosted ensemble (current)

Step 3

What you should see

The first tree is just a step function. Depth-3 means at most 8 leaves; the prediction is a blocky approximation of the truth.
Each subsequent tree corrects the residual. Watch the residual panel: it gets noisier and smaller, never really hitting zero unless you keep adding trees.
Lower η = smoother fits. At η=0.05 + 100 trees the curve is smooth. At η=0.5 + 20 trees it's jagged. Same total "step size", different bias/variance.
Boosted vs RF. Random forest averages many independent deep trees; boosting builds dependent shallow trees. RF is robust + parallel; boosting is more accurate + sequential. For tabular tasks XGBoost / LightGBM / CatBoost beat RF on most benchmarks since 2018.

Step 4

Production XGBoost — what to actually tune

n_estimators / num_boost_round. Use early stopping with a validation set. 500–5000 is the useful range; more is wasted.
learning_rate (η). 0.01–0.1 for big datasets, 0.05–0.3 for small. Lower with more trees.
max_depth. 3–10. Deeper for big tabular data with rich interactions; shallower for noisy data.
min_child_weight. Equivalent of min-samples-per-leaf weighted by the Hessian. 1 default; push higher for noisy data.
subsample / colsample_bytree. Bootstrap row / feature sampling per tree. 0.7–0.9 is the sweet spot — adds randomness, regularises.
reg_alpha / reg_lambda. L1 / L2 on leaf weights. Tune if you have many features and worry about overfitting.
tree_method. Use 'hist' (default in modern XGBoost) for everything; it's faster and as accurate as 'exact'.
objective. 'reg:squarederror', 'binary:logistic', 'multi:softprob', 'rank:pairwise', 'reg:tweedie' (zero-inflated counts), …
monotonic constraints. Force "prediction must be monotone in feature X" — useful for insurance, healthcare, regulation.
missing values. XGBoost natively learns best-direction routing for NaN per split; don't impute unless you must.

The slogan. For tabular data with mixed types, missing values, and $\le$ a few million rows, XGBoost / LightGBM / CatBoost are still the strongest and simplest default. Neural-tabular models (TabNet, FT-Transformer, SAINT) win benchmarks occasionally; gradient-boosted trees still win most of the production time.

Step 5

Variants worth knowing

LightGBM. Histogram-based; leaf-wise (deepest-first) splitting; faster than XGBoost on big data; slightly more overfit-prone, easily controlled with num_leaves and regularisation.
CatBoost. Native handling of categorical features (target encoding without leakage); excellent default hyperparameters; slowest of the three on large data.
HistGradientBoosting (sklearn). Pure-Python alternative; much faster than the original sklearn GBM; comparable accuracy; ships with sklearn.
NGBoost. Gradient boosting that outputs a distribution (mean + variance); useful for downstream uncertainty work.
XGBoost-Spark / Dask-XGBoost. Distributed for very large data.
TabNet, FT-Transformer, SAINT. Neural tabular alternatives; sometimes win, often tie, rarely justify the extra ML-ops complexity.