Interactive Explainer

Vanishing & Exploding Gradients

Stack a hundred layers; multiply a hundred Jacobians; watch the gradient die — or blow up. A derivative panel demystifies the magic factor (0.25 for sigmoid, 1.0 for ReLU); a depth slider shows the multiplicative chain compound; a weight-scale slider flips the failure mode from vanishing to exploding. Then a misconceptions card cleans up what "vanishing" really means.

~14 min Deep learning · Foundations · Initialization

Prelude

Why gradients matter at all

A neural network learns by subtracting a tiny multiple of $\partial \mathcal{L} / \partial W$ from each weight. If that gradient is essentially zero, the weight doesn't move and the layer doesn't learn. If it's astronomical, the weight overshoots and the loss explodes. Both failure modes have the same root cause: the gradient at layer 1 is a product of $L$ factors — one per layer downstream of it.

A product of numbers all less than 1 shrinks to zero. A product of numbers all greater than 1 blows up. The two failures — vanishing and exploding — are the same algebraic accident, with opposite signs.

The interactives below take you through the story in three layers: (1) what a single activation's derivative actually looks like; (2) what stacking the chain does; (3) what changing weight scale or activation function buys you.

Step 1

Where the magic factor 0.25 comes from

Every backprop chain factor is the derivative of the layer's activation function evaluated at the pre-activation $z$. So we should start by looking at that derivative. Drag the input $z$ below; the dot on the left panel walks the activation function, and the dot on the right panel walks its derivative. The right panel is the one that ends up in the chain.

activation:

z 0.00

activation σ(z)

derivative σ'(z) — this is what backprop multiplies

σ'(z) at current slider: 0.250

max σ'(z) over all z: 0.250

Why sigmoid is doomed at depth: even at its most generous point ($z = 0$), $\sigma'(z) = 0.25$. Anywhere else, it's smaller. The chain rule is going to multiply $L$ of these numbers together — and they are all at most $0.25$.

Sigmoid. $\sigma'(z) = \sigma(z)(1 - \sigma(z))$. Max value $0.25$ at $z = 0$; falls off to zero on either side.
Tanh. $\tanh'(z) = 1 - \tanh(z)^2$. Max value $1.0$ at $z = 0$ — generous in the middle, saturates fast outside $|z| > 2$.
ReLU. $\sigma'(z) = \mathbb{1}\{z > 0\}$ — either exactly 1 (active) or exactly 0 (dead). No saturation; a discrete switch.

Step 2

The chain compounds — let's watch it

Now stack $L$ identical layers, each contributing the activation's best-case derivative to the chain. The input-layer gradient is the cumulative product of those $L$ factors. Below: the bar at the top is the output layer (where backprop starts); each bar below it is one more multiplication into the chain. The bottom bar is the input layer — the one whose gradient we care about when we say "vanishing".

activation:

depth L 12

input-layer gradient

—

per-layer factor

—

depth at 1e-9

—

gradient > 10⁻³ (alive) 10⁻⁶ – 10⁻³ (shrinking) < 10⁻⁶ (effectively zero) > 10² (exploding)

log₁₀ |gradient|: 10⁻²⁰ ← shrunk · alive → 10²

the multiplicative chain (input-layer gradient written out)

—

Things worth trying:

Set activation to sigmoid, depth $L = 10$: the chain is $0.25^{10} \approx 9.5 \times 10^{-7}$ — already in the "shrinking" band.
Sigmoid at $L = 20$: $0.25^{20} \approx 9 \times 10^{-13}$ — below typical float-32 precision. The layer cannot learn.
Switch to tanh: max derivative 1.0; at $L = 50$ the gradient stays at $1.0^{50} = 1$. But that's the best case — at any $|z| > 2$, $\tanh' \ll 1$ and the real chain still vanishes, just more slowly.
Switch to ReLU with active fraction 100%: gradient stays at 1 forever. This is why ReLU broke the depth ceiling.

Step 3

The other failure mode: exploding

We have been ignoring the weight matrices in the chain. The real per-layer factor is $W_\ell^\top \sigma'(z_\ell)$ — the activation derivative times the weight. If we initialise the weights too large, the product is greater than 1, and the chain blows up instead of dying.

advanced: weight initialisation scale (try this)

Move the slider below. Each layer now contributes $w \cdot \sigma'(z)$ to the chain. Values $w > 1$ flip the failure mode from vanishing to exploding.

weight scale w 1.00 activation depth 20 ReLU active fraction 50%

input-layer gradient

—

per-layer factor (w · σ')

—

regime

—

The five fixes the field converged on. Every one of them is targeted at keeping the per-layer chain factor close to 1.

ReLU and friends (Glorot & Bengio 2010; Nair & Hinton 2010): $\sigma' \in \{0, 1\}$ instead of $\le 0.25$.
Xavier / He initialisation: scale weights so $\mathrm{Var}(W_\ell) \cdot \mathbb{E}[\sigma'^2] = 1$ — the chain factor's variance is 1.0 by construction.
Batch / Layer norm: rescale activations layer-by-layer so the per-layer Jacobian's spectral radius stays near 1.
Residual connections (He et al. 2015): $y = f(x) + x$. The identity path adds 1 to the Jacobian; even if $f$ vanishes, $1$ does not.
LSTM cell state / gated cells: the forget gate $\in [0, 1]$ multiplies the cell-state Jacobian directly; setting it near 1 gives a gradient highway across thousands of timesteps.

Step 4

Three misconceptions, three corrections

"Dying ReLU neurons cause vanishing gradients."

Not quite. Vanishing gradients from sigmoid / tanh are multiplicative — the effect compounds exponentially with depth ($0.25^L$). A 50% ReLU-neuron death rate is a linear scaling — $1^L \cdot 0.5 = 0.5$ at every depth. It shrinks once, but it doesn't vanish exponentially. The two are different problems with different fixes.

"A vanishing gradient makes the weights go to zero."

The updates go to zero, not the weights: $W_{\text{new}} = W_{\text{old}} - \eta \cdot 0 = W_{\text{old}}$. The weights stay at their random initialisation; the layer fails to learn anything meaningful, but it isn't pushed toward zero.

"This (factor)^N model is just a cartoon."

It's a powerful illustration, not a precise simulation. In a real network every $\sigma'(z_\ell)$ is different. But because the sigmoid derivative is always $\le 0.25$ and tanh's is $\le 1.0$, the worst-case bound $0.25^L$ governs the product. The cartoon captures the asymptotic behaviour exactly; the constants are optimistic.

"Vanishing gradients are a sigmoid problem; ReLU solved them."

ReLU made vanishing much less common, but not impossible. A network with $w < 1$ initial weights and a non-residual architecture can still see gradients shrink layer by layer — the chain is $w^L$ rather than $0.25^L$, but the principle is identical. Modern training relies on initialisation and normalisation and residual paths together.

Step 5

Numbers to remember

Six facts that make the next time you look at a deep network's training curve immediately legible:

$\sigma'_{\max} = 0.25$ (sigmoid), $1.0$ (tanh), $1$ (ReLU active).
Sigmoid hits the gradient floor at depth $\approx \log_4 (1 / \text{floor}) \approx 20$ for floor $= 10^{-12}$.
Xavier init scales weights as $\mathcal{N}(0, 1/\text{fan\_in})$; He init scales as $\mathcal{N}(0, 2/\text{fan\_in})$. The factor 2 is because ReLU kills half the activations on average.
BatchNorm typically rescales layer outputs to $\mathrm{Var} = 1$; this is equivalent to "renormalise the Jacobian's spectrum to 1" at every layer.
Residual connections add an identity to the Jacobian, so the lower bound on the gradient is $1$, not $0$.
Gradient clipping caps the exploding direction; it doesn't fix vanishing. Use clipping for RNNs / transformers training stability; don't expect it to make a sigmoid-MLP train at depth 50.

Step 6

Reading list

Hochreiter, 1991 (Diploma thesis) — the original "long-term dependencies are hard" analysis.
Bengio, Simard, Frasconi, 1994 — Learning long-term dependencies with gradient descent is difficult. The formal vanishing-gradient analysis.
Glorot & Bengio, 2010 — Understanding the difficulty of training deep feedforward neural networks. Xavier init; the modern story starts here.
Nair & Hinton, 2010 — Rectified linear units improve restricted Boltzmann machines.
He et al., 2015 — Delving deep into rectifiers. He init; ReLU-aware variance.
He et al., 2015 — Deep residual learning for image recognition. Residual connections.
Pascanu, Mikolov, Bengio, 2013 — On the difficulty of training recurrent neural networks. Exploding gradients and clipping.
Karpathy, "Yes you should understand backprop" — readable companion piece arguing students should know all of this without autograd.