Residual Connections, Why They Work

Prelude

The depth degradation problem

Before December 2015, conventional wisdom said deeper networks should generalise better — until you tried to train a plain 50-layer CNN and watched it do worse than the 20-layer version on the training set, never mind the test set. That's not overfitting; it's a failure of the optimiser. He et al. called it the degradation problem.

The fix was a single line of code: change every block from $y = F(x)$ to $y = F(x) + x$. The output is now whatever the block computes plus a copy of the input the block could ignore. That tiny change made it possible to train ResNet-152, ImageNet-class accuracy at 3× the depth of every prior architecture, and started the modern deep-learning rocket.

The slogan. A residual block doesn't have to learn the right output. It only has to learn the residual — the small correction on top of the input. If a layer has nothing to add it can output zeros, and the input passes through untouched.

Step 1

Watch activations die without skips

Below: a tiny feed-forward stack at depth $L$. Each layer is $h_{l+1} = \tanh(W_l h_l + b_l)$, randomly initialised. The plain version is just that recursion. The residual version replaces it with $h_{l+1} = h_l + \tanh(W_l h_l + b_l)$. Slide $L$ and watch the activation magnitude at each depth.

depth L 32

init scale 0.6

Activation L₂-norm per layer. Solid teal: residual stack. Dashed orange: plain stack. The plain trace decays to zero (or blows up); the residual stack stays bounded.

Step 2

Gradient flow — the same story, backwards

Backpropagation computes $\frac{\partial \mathcal{L}}{\partial h_l}$ as a chain of matrix products. In a plain net every link in the chain is some Jacobian close to a non-identity matrix; their product collapses fast. In a residual net the Jacobian is $I + \frac{\partial F}{\partial h_l}$ — identity plus something small. Identity preserves; "plus something small" adds gradient signal layer by layer instead of multiplying it away.

Below: gradient norm at each depth, traced from the loss back to layer 0. The residual line is roughly flat. The plain line falls off geometrically.

‖∂ℒ/∂h_l‖ vs depth l, on a log scale.

Step 3

Train both, side by side

Same dataset, same depth, same optimiser. Press Start training below; one canvas trains the plain net, the other trains the residual net. The plain version visibly stalls at $L \ge 16$; the residual version trains cleanly to much greater depth.

depth L 12

learning rate 0.05

step 0 plain loss — residual loss —

Training loss for the two architectures, same data, same lr. Crank L up to 64 and watch the gap blow open.

Step 4

The loss surface, visualised

Li et al. (2018) showed that residual connections smooth the loss landscape. We render a 2-D slice: pick two random directions in parameter space, evaluate the loss on a grid, and contour. The plain landscape is ridged and full of bad minima; the residual landscape is a single shallow bowl.

Plain net (L = 24)

Residual net (L = 24)

Lighter regions are lower loss. Notice the plain surface has multiple basins separated by ridges; the residual surface has a single broad basin around the origin. Same model class, one line of code different.

Step 5

Iterative refinement — what each block does

A useful mental model (Greff et al., Jastrzębski et al., Veit et al.): each residual block makes a small correction to a running representation. The "residual stream" carries the input forward unchanged; blocks are read-write peripherals that nudge it.

That's also exactly the picture of a Transformer's residual stream — attention and MLP blocks read the stream, add their contributions, and pass it on. Mechanistic interpretability work (Elhage et al., 2021) treats the stream as a literal communication channel between layers.

What the model can do that a plain stack cannot. A residual stack can choose its own depth: any unnecessary block just outputs $\approx 0$ and the input passes through. A plain stack has to use every layer fully — and the optimiser has to learn an identity subnetwork from scratch to do so.

Step 6

The variants and their lessons

ResNet-v1. $y = \mathrm{ReLU}(F(x) + x)$. Original, still strong on ImageNet.
Pre-activation ResNet. $y = F(x) + x$ with the BN→ReLU before the conv. Cleaner gradient flow; preferred for very deep stacks (1000+ layers).
DenseNet. Concatenate every previous output instead of summing. More parameter-efficient, slightly slower on hardware due to memory.
Highway / gated residuals. $y = T(x) \odot F(x) + (1 - T(x)) \odot x$ with a learned gate $T$. ResNet is the special case $T = 1$. Recovers a similar idea in LSTMs.
Transformer residuals. Every attention and MLP block reads-and-writes a residual stream. The mathematical justification is identical to ResNet's.

Final takeaway. One line of code — "and add the input back" — was the deep-learning equivalent of the printing press. Without residuals, no ResNet-152, no BERT, no GPT-anything. Every time you stack more than ~10 layers and it just works, you're using this trick.

Step 7

Why the residual stream view unlocks Transformers

Modern transformers are best understood as residual streams (Elhage et al., 2021, Anthropic interpretability). Each layer reads from a high-dimensional residual stream, computes a delta, and writes it back. Layer $\ell$'s output is

Three consequences of this view:

Layers communicate through the stream. Layer 12 can read what layer 3 wrote because the residual stream carried it through unchanged (modulo whatever later layers added).
The stream is a "shared workspace". Different heads / MLPs specialise in writing different "kinds" of information; downstream layers read what's relevant. Mech-interp papers identify "induction heads", "successor heads", "letter-counting circuits" that span multiple layers.
LayerNorm is the gate. Pre-norm transformers (norm before attention/MLP) work better than post-norm precisely because pre-norm preserves the residual identity path. Post-norm transformers need warmup; pre-norm don't.

Step 8

Where residuals can hurt

Three failure modes to know:

Representation rank collapse. Stacking many residual blocks with similar deltas can push the residual stream into a low-rank manifold (Dong et al., 2021). Symptoms: late-layer attention heads all attending to the same token, slow convergence. Fix: stronger regularisation, depth scaling, or initialisation that biases each block to write something distinct.
"Free path" overuse. If a block consistently outputs near-zero, you've wasted parameters; if every block outputs strongly, the residual stream is dominated by noise. Healthy transformers sit in between, but it's worth checking layer-output norms during training.
Skip-connection memory cost. The original input has to be kept around through the block for the addition; this doubles activation memory in some implementations. Reversible variants (RevNet) recompute activations on the backward pass; rarely needed in practice.

Step 9

Reading list

He, Zhang, Ren, Sun (2015) — Deep Residual Learning for Image Recognition. The ResNet paper.
He et al. (2016) — Identity Mappings in Deep Residual Networks. Pre-activation variant.
Huang et al. (2017) — DenseNet.
Veit, Wilber, Belongie (2016) — Residual Networks Behave Like Ensembles of Relatively Shallow Networks. The "exponential ensemble of paths" interpretation.
Elhage et al. (2021) — A Mathematical Framework for Transformer Circuits. The residual stream view.
Dong, Cordonnier, Loukas (2021) — Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth.