Interactive Explainer

Box Regression Losses

A detector predicts four numbers per box — position and size — and trains them by penalizing the difference from the ground truth. But how you penalize that difference matters. L2 obsesses over the worst box; L1 treats every error the same but has a kink at zero; smooth-L1 blends the two. Drag the predicted box and watch the three losses argue.

~8 min Deep Learning · Object Detection · Loss Functions

Prelude

Four numbers, one residual vector

A predicted box is a vector $\hat{\mathbf{b}}=(x,y,w,h)$ and the target is $\mathbf{b}^{*}=(x^{*},y^{*},w^{*},h^{*})$. The regression loss is a sum over the four per-coordinate residuals $r_i = \hat{b}_i - b^{*}_i$:

\mathcal{L}(\hat{\mathbf{b}},\mathbf{b}^{*}) = \sum_{i} \ell(r_i), \qquad r_i = \hat{b}_i - b^{*}_i .

Everything hinges on the shape of the per-coordinate penalty $\ell(r)$. The three classic choices:

\ell_{1}(r) = |r|, \qquad \ell_{2}(r) = \tfrac{1}{2}r^{2}, \qquad \ell_{\text{smooth}}(r) = \begin{cases} \frac{r^{2}}{2\beta}, & |r| < \beta \\[2pt] |r| - \frac{\beta}{2}, & |r| \ge \beta \end{cases}.

The Lab

Move the box, read the loss

The green dashed box is the fixed ground truth. The blue box is your prediction — move it with the sliders. The right panel plots the three per-coordinate penalties; the sliding marker sits at the current $x$-residual $r_x$, so you can watch each loss respond as you push the box off target.

Pred $x$ 3.10

Pred $y$ 2.10

Pred $w$ 2.60

Pred $h$ 2.10

Smooth-L1 transition $\beta$ 1.00

(a) Predicted vs. ground-truth box. Arrows show the four residuals $r_x,r_y,r_w,r_h$.

(b) Per-coordinate penalty vs. residual $r$. The dot marks $r_x$ on each curve.

L1 $|r|$ L2 $\tfrac12 r^2$ smooth-L1

L1 loss $\;\sum|r_i|$ —

L2 loss $\;\sum \tfrac12 r_i^2$ —

Smooth-L1 $\;\sum \ell_\beta(r_i)$ —

Push the box far off target. The L2 number rockets up quadratically while L1 and smooth-L1 grow only linearly. Now flip to Show gradients: L2's gradient is the straight line $r$, so a single wildly-wrong box produces a huge gradient that swamps the update for all the well-placed boxes. That outlier sensitivity is exactly what smooth-L1 was designed to fix.

The Payoff

Why smooth-L1 wins for boxes

Read the three gradients side by side — the gradient is what actually trains the network:

\ell_{2}'(r) = r, \qquad \ell_{1}'(r) = \operatorname{sign}(r), \qquad \ell_{\text{smooth}}'(r) = \begin{cases} r/\beta, & |r| < \beta \\ \operatorname{sign}(r), & |r| \ge \beta \end{cases}.

L2 has gradient $r$: smooth and well-behaved near the target, but it grows without bound. One badly-initialized box early in training can dominate the batch.
L1 has gradient $\pm 1$: bounded and robust to outliers, but it never eases off. Even a near-perfect box keeps getting a full-strength $\pm 1$ nudge, and the kink at $r=0$ makes the gradient discontinuous.
Smooth-L1 is $r/\beta$ near zero (like L2, so it settles gently and smoothly through the origin) and $\pm 1$ in the tails (like L1, so outliers cannot blow up). The transition happens at $|r|=\beta$.

This is why detectors use it. Fast R-CNN introduced smooth-L1 for exactly this reason, and the pattern carried into Faster R-CNN, SSD, and RetinaNet. Box targets are noisy and some anchors start far away; the quadratic core keeps optimization stable once boxes are close, while the linear tails stop a handful of terrible boxes from hijacking the gradient. Move $\beta$ to shift where robustness takes over from precision.