Interactive Explainer

Weight Decay Shrinks the Fit

Hand a degree-9 polynomial a fistful of noisy points and it will happily thread every one — wiggling violently between them with enormous coefficients. Weight decay adds one term, $\lambda\lVert w\rVert^{2}$, that taxes big weights. Turn the $\lambda$ knob below and watch the wild curve calm down, the coefficients collapse, and the test error slide into its sweet spot.

~10 min Deep Learning · Generalization · Regularization

Prelude

One extra term in the loss

Ordinary least squares picks weights $w$ that minimize the squared error on the training data. Ridge regression — the linear-model name for weight decay — adds a penalty proportional to the squared length of $w$:

\mathcal{L}(w) = \underbrace{\sum_{i=1}^{N}\big(y_i - w^{\top}\phi(x_i)\big)^2}_{\text{fit the data}} \;+\; \underbrace{\lambda\,\lVert w\rVert^{2}}_{\text{keep }w\text{ small}} .

Because this is quadratic in $w$, there is a closed-form minimizer — no gradient descent required. Setting the gradient to zero gives the ridge normal equations, which this page solves live in your browser with a tiny Gaussian-elimination routine:

\big(\Phi^{\top}\Phi + \lambda I\big)\,w = \Phi^{\top}y \qquad\Longrightarrow\qquad w^{*} = \big(\Phi^{\top}\Phi + \lambda I\big)^{-1}\Phi^{\top}y .

Here $\Phi$ is the design matrix whose rows are the polynomial features $\phi(x) = [1, x, x^2, \dots, x^d]$. The single knob $\lambda$ interpolates between two extremes: at $\lambda \to 0$ you recover the wild interpolating fit; as $\lambda \to \infty$ every weight is crushed toward zero and the curve flattens to a constant.

The Lab

Turn the knob, watch the fit

Fifteen noisy points (orange) are sampled from a smooth true curve (grey dashed). A degree-9 polynomial is fit by ridge regression. Drag $\lambda$ on its log scale: far left is almost-zero penalty (wild overfit), far right is a heavy penalty (flat underfit). The three presets jump you to the interesting regimes.

Penalty $\lambda$ = 0.010

Polynomial degree $d$ = 9

Noise level $\sigma$ = 0.18

The fitted degree-$d$ polynomial (blue) against the true function (grey dashed) and the noisy training sample (orange). A vertical exaggeration is clipped to the frame.

Weight norm $\lVert w\rVert^{2}$ —

Train error (MSE) —

Test error (MSE) —

$\lVert w\rVert^{2}$ versus $\lambda$ (both log axes). Bigger penalty → smaller weights, monotonically.

Train (blue) vs test (green) error versus $\lambda$. Test error is a U — the dot marks your current $\lambda$.

The learned coefficients $w_1,\dots,w_d$ (the bias $w_0$ is not penalized and is hidden). Watch every bar collapse toward zero as $\lambda$ grows.

Start at "Overfit." The blue curve snakes through nearly every orange dot, but between them it lunges toward $\pm\infty$; the weight-norm number is astronomical and the test error is far above the train error. Now drag $\lambda$ right: the curve relaxes onto the true grey curve, the coefficient bars shrink, and test error drops into the trough before the curve finally goes flat.

The Payoff

Why smaller weights generalize better

A high-degree polynomial is not too complex because it has many terms — it is dangerous because those terms can take on huge, canceling values. A coefficient of $+40{,}000$ on $x^7$ fighting $-38{,}000$ on $x^9$ is what produces the violent wiggles. Penalizing $\lVert w\rVert^{2}$ forbids exactly those enormous, delicately balanced weights, so the only fits left are smooth ones.

Bias–variance, made visible. Small $\lambda$ = low bias, high variance (fits the noise). Large $\lambda$ = high bias, low variance (ignores signal too). The U-shaped test curve is that trade-off, and its minimum is the $\lambda$ you want.
Same math as an MAP prior. $\lambda\lVert w\rVert^{2}$ is the log of a Gaussian prior $w \sim \mathcal{N}(0, \tfrac{1}{2\lambda}I)$. Weight decay is just Bayesian shrinkage toward zero.
Why "decay." In gradient descent the penalty adds $-2\eta\lambda\,w$ to each step, i.e. every weight is multiplied by $(1-2\eta\lambda)<1$ before the data-driven update — it literally decays toward zero each step.

In deep nets. There is no closed form, but the idea is identical: add $\lambda\lVert w\rVert^{2}$ to the loss (or use an optimizer's weight_decay argument). It is one of the cheapest, most reliable regularizers there is — a single number that buys you a smoother function and a smaller gap between training and test performance.