Interactive Explainer

Watch an RNN's State Evolve

A recurrent network is just one small cell applied over and over. It keeps a running hidden state $h_t$, and at every timestep it folds the next input into that state with the same weights. Set the weights, feed a sequence, and press Step to watch a whole sequence collapse into a single number, one timestep at a time.

~8 min Deep Learning · Sequences · RNNs

Prelude

One cell, applied over and over

Take the simplest possible recurrent unit: a single scalar hidden state $h_t$. At each step it reads one input $x_t$, mixes it with the previous state, and squashes the result through $\tanh$:

h_t = \tanh\!\big(w_x\,x_t + w_h\,h_{t-1} + b\big), \qquad h_0 = 0 .

There are only three numbers to learn — $w_x$ (how strongly the new input matters), $w_h$ (how strongly the past matters), and a bias $b$ — and they are shared across every timestep. That weight sharing is the whole idea: a sequence of any length is processed by the same tiny function, and all the network's memory lives in the single running number $h_t$.

The Lab

Step the recurrence by hand

Choose an input sequence, set the three weights, and press Step. Each press advances $t$ by one and shows the exact arithmetic with the numbers substituted in — then plots the new state on the trajectory. Watch how the recurrent weight $w_h$ decides whether the state settles, oscillates, saturates, or (in linear mode) blows up.

Input sequence:

Or type your own (comma-separated)

Input weight $w_x$ = 0.80

Recurrent weight $w_h$ = 1.10

Bias $b$ = 0.00

Activation

Hidden state $h_t$ against timestep $t$. Filled dots are revealed steps; the faint dashed line previews where the current weights are heading. Grey stems at the bottom show the input $x_t$.

Current step $t$ 0

Current state $h_t$ 0.000

Final state $h_T$ —

Push $w_h$ toward 2 and step through the Step input. In tanh mode the state races to $+1$ and pins there — it saturates, and any further input barely moves it. Switch to linear mode with the same $w_h > 1$ and the state instead doubles every step: it explodes. That single knob, $w_h$, is the entire vanishing / exploding story in miniature.

The Payoff

A sequence, folded into one number

Nothing above ever stored the sequence. At step $t$ the cell only ever sees $x_t$ and the single carried number $h_{t-1}$ — yet $h_t$ depends on the whole history $x_1,\dots,x_t$, because each state was built from the one before it. That is what “recurrent” means: memory is a running summary, not a stored transcript.

With $|w_h| < 1$ the influence of an old input decays geometrically — the network has a short memory and forgets the impulse.
With $w_h \approx 1$ the state holds information for a long time — the sweet spot, but a knife-edge that is hard to learn.
With $|w_h| > 1$ the state saturates (through $\tanh$) or explodes (linear) — the same instability that makes deep RNNs hard to train.

Where this goes next. Because $h_t$ is built by re-applying one cell, training it means differentiating through that entire chain — backprop through time. And because every step multiplies by roughly $w_h$, gradients shrink or grow exponentially with sequence length. LSTMs and GRUs exist precisely to give the state an easier, gated path through time.