Interactive Explainer

Backprop Through Time

To train an RNN you unroll it into a deep feed-forward graph with shared weights, then run ordinary backprop across it. Walk the backward pass step by step: at each state, watch its gradient accumulate two pieces — one from the loss right here, one carried back from the future — and see how truncation cuts the chain and how the per-step product quietly vanishes or explodes.

~9 min Deep Learning · RNNs · Backpropagation

Prelude

Unroll, then backprop

An RNN reuses one cell at every timestep, so in time it looks like a very deep network whose layers all share the weights $w_h, w_x, b$. Draw it unrolled and it is just a chain: inputs $x_t$ come in from below, an output/loss $\mathcal{L}_t$ is read off above, and the hidden state is threaded left to right.

Backprop on this graph has a name — backpropagation through time. Its one wrinkle: every hidden state feeds two places, so its gradient is a sum of two incoming contributions.

\frac{\partial \mathcal{L}}{\partial h_t} = \underbrace{\frac{\partial \mathcal{L}_t}{\partial h_t}}_{\text{local: loss here}} \;+\; \underbrace{\frac{\partial \mathcal{L}}{\partial h_{t+1}}\,\frac{\partial h_{t+1}}{\partial h_t}}_{\text{carried back from the future}} .

The Lab

Walk the backward pass

Seven timesteps, unrolled. Press Step backward to move the gradient frontier from the last step toward the first. At each state the two contributions light up: the vertical arrow is the local loss gradient, the horizontal arrow is credit carried back from $h_{t+1}$. The truncation $K$ slider severs the chain — greyed states are cut off, and no credit ever reaches them.

Truncation length $K$ = 4 steps

Recurrent weight $|w_h|$ = 0.85

Activation slope $\sigma'$ = 0.90

The RNN unrolled over 7 timesteps. Blue = forward pass. Orange = gradients flowing backward. Faded states are severed by truncation $K$.

forward pass local loss gradient credit carried back severed by truncation

Backward step 0 / 4

Gradient frontier at —

Credit reaching frontier —

Press Step backward. Watch the frontier march left and the "credit reaching frontier" number change. Each hop multiplies it by $|w_h|\cdot\sigma'$ — set that product below 1 and credit vanishes as it travels back; set it above 1 and it explodes.

The Payoff

Why long-range credit is so fragile

Trace the credit that reaches an early state $h_t$ from a late loss $h_T$. It has to pass through every recurrent edge in between, and each edge contributes the same Jacobian factor. The whole path is therefore a product:

\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}} \approx \big(w_h\,\sigma'\big)^{\,T-t} .

If $|w_h\,\sigma'| < 1$, that product decays exponentially — the vanishing gradient. Distant timesteps get essentially zero learning signal.
If $|w_h\,\sigma'| > 1$, it grows exponentially — the exploding gradient, which gradient clipping papers over but never truly fixes.
Only $|w_h\,\sigma'| \approx 1$ preserves credit over long ranges — a razor's edge that plain RNNs cannot reliably sit on.

Truncated BPTT is the practical compromise. Instead of unrolling the entire sequence, you backprop only $K$ steps and cut the rest (the greyed region). It bounds compute and memory per update and tames exploding gradients — at the cost that no gradient signal ever reaches beyond $K$ steps, so the model simply cannot learn dependencies longer than its truncation window.

The fix that stuck. LSTMs and GRUs replace that fragile $w_h\,\sigma'$ product with a nearly-linear, gated highway for the state, so the per-step factor stays close to 1 by construction. Same unrolled picture — just an easier road for credit to travel.