Double Descent, Made Concrete

What you should see

• Under-parameterised regime ($P < N$). Train and test error both drop as you add parameters — classical bias-variance.
• Interpolation threshold ($P \approx N$). Train error hits zero; test error spikes. The model just barely fits the training data and pays for it with wild oscillations.
• Over-parameterised regime ($P \gg N$). Past the spike, test error descends again. This is what modern DL has been quietly relying on.

The mechanism (Belkin et al., 2018; Hastie et al., 2019): past the threshold, infinitely many weight vectors achieve zero training error, and the optimiser picks the minimum-$\ell_2$-norm one. Many of those are smooth, and the implicit bias of gradient descent (and ridge as $\lambda \to 0^+$) finds them.

Why classical bias-variance predicted the wrong shape

The U-curve from any ML textbook decomposes test error as

Up to the threshold, variance dominates the rise. The textbook stops here. What the textbook missed is that past $P = N$, both bias and variance can drop because the model now has slack capacity: many feature directions encode noise, and the minimum-norm solution spreads its weights across them so each individual direction carries less of the noise — the variance is averaged down. Belkin et al. coined this the "modern interpolating regime" in 2018.

Live curve

Why the spike happens, in three lines of algebra

With random feature matrix $\Phi \in \mathbb{R}^{N \times P}$ and tiny ridge $\lambda$, the prediction at a test point is $\hat y_* = \phi(x_*)^\top \hat w$. As $P \to N$, the smallest singular value $\sigma_{\min}(\Phi)$ approaches zero, and the ridge inverse blows up:

Small $\sigma_i$ correspond to feature directions that the training data barely identifies. Without ridge, those directions soak up huge weight to fit the noise — variance explodes. The spike happens exactly at $P = N$ because that's where $\Phi$ becomes square and least-squares is forced to interpolate. Beyond $P = N$, the system is underdetermined; min-norm collapses those weights again.

Stronger ridge ($\lambda \uparrow$) flattens the spike. Slide ridge log₁₀λ up and watch the peak shrink — that's the variance term being dampened. The cost: slightly higher error far from the threshold.

Live: sample-wise double descent

Fix the model size $P$ above the original interpolation threshold. Now sweep the training set size $N$ from small to large. As $N$ approaches $P$ from below, the model is forced out of the over-parameterised regime back through the threshold. Test error can increase with more training data — exactly the opposite of what every intro ML class teaches.

Nakkiran et al. (2019, "Deep Double Descent") observed this same shape in CIFAR ResNets when sweeping the training set size with the model fixed. The remedy is more weight decay or much more training data.

Epoch-wise double descent

A third axis: hold $P$ and $N$ fixed, watch test error vs training epoch. Long-running ResNet runs show test error first drop, rise around the epoch where training loss hits zero, then drop again. The mechanism is the same — the network passes through an effective interpolation threshold as its function class "fills out" during training.

What it means in practice

• Don't be afraid of overparameterisation. For deep nets, "more capacity" is usually safer than "less capacity" provided you have a sane regulariser (weight decay, dropout, early stopping).
• The spike is real. Training a CIFAR ResNet at the exact interpolation threshold consistently underperforms either smaller or much larger nets at the same compute budget.
• Sample-wise double descent. Adding training data can sometimes increase test error if it pushes the model from over-parameterised back to interpolating exactly. Real phenomenon (Nakkiran et al., 2019).

Connection to the NTK and infinite width

The Neural Tangent Kernel limit (Jacot et al., 2018) says that an infinitely-wide net trained with gradient descent and vanishing learning rate behaves as a fixed kernel regressor. The "$P = N$" spike is where that kernel's training-data Gram matrix becomes ill-conditioned; the second descent corresponds to the kernel ridgeless limit on an effectively over-determined system.

This gives a clean explanation:

• Under the NTK, "more parameters" = more random features.
• The spike at $P = N$ is exactly the random-feature ridge spike we just plotted.
• Real neural nets also learn features, so the real shape is a mixture of this and a feature-learning descent on top.

Hastie, Montanari, Rosset & Tibshirani (2019) give the asymptotic risk formula for random-feature ridge regression in closed form: the test risk peaks at exactly $P/N = 1$ for $\lambda \to 0$ and decreases monotonically thereafter, with the rate of descent governed by the spectrum of the feature map.

How to avoid getting trapped at the spike

• Weight decay (or any $\ell_2$). Almost erases the spike. Cost: very mild test-error increase away from threshold. Practically, this is the default.
• Don't size your model at $P \approx N$. If you're scaling a model deliberately, either stay clearly under the threshold or clearly over it. The danger zone is narrow but real.
• Early stopping reduces epoch-wise descent by cutting training before the model reaches the interpolating epoch. Less common in modern LLM pretraining (which trains long); essential for small-data fine-tunes.
• Data augmentation shifts the threshold by inflating $N$. A solid practical fix that costs nothing in architecture changes.
• Label noise warning. Noisy labels make the spike taller and wider. Clean labels first; everything else helps less.

Reading list

• Belkin, Hsu, Ma, Mandal (2018) — Reconciling modern machine-learning practice and the classical bias–variance trade-off. The paper that named "double descent" and connected modern DL to classical kernel theory.
• Nakkiran, Kaplun, Bansal, Yang, Barak, Sutskever (2019) — Deep double descent: where bigger models and more data hurt. The empirical survey on ResNets / Transformers; defined epoch-wise and sample-wise variants.
• Hastie, Montanari, Rosset, Tibshirani (2019) — Surprises in high-dimensional ridgeless least squares interpolation. The clean theoretical paper for random-feature ridge regression with proofs.
• Bartlett, Long, Lugosi, Tsigler (2020) — Benign overfitting in linear regression. Gives precise conditions on the covariance spectrum under which interpolation generalises.