Interactive Explainer

GRU Gates: Keep or Update

A GRU cell has one job at each step: decide how much of the old state to keep and how much to replace with something new. That decision is a learned interpolation. Slide the update gate $z$ and watch the new state glide between "remember the past" and "take the candidate" — while the reset gate $r$ controls how much of the past the candidate is even allowed to see.

~8 min Deep Learning · RNN · GRU

Prelude

Two gates, one interpolation

Where a plain RNN overwrites its state every step, a GRU protects it with two sigmoid gates. The reset gate $r$ decides how much of the past leaks into a fresh candidate state $\tilde{h}_t$; the update gate $z$ then blends that candidate with the old state:

\tilde{h}_t = \tanh\!\big(W_x x_t + r_t \odot W_h h_{t-1}\big), \qquad h_t = (1 - z_t)\,h_{t-1} + z_t\,\tilde{h}_t .

Read that second equation as a dial. When $z_t = 0$ the cell copies the past through untouched ($h_t = h_{t-1}$); when $z_t = 1$ it throws the past away and adopts the candidate; anywhere in between it is a weighted average. Gates are just learned interpolators.

The Lab

Turn the dials

Set the previous state $h_{t-1}$ and an input $x_t$, then move the two gates. The candidate box updates its arithmetic as you change $r$; the number line shows the new state $h_t$ sliding along the segment between the past and the candidate at exactly fraction $z$.

Presets:

Update gate $z$ = 0.50

Reset gate $r$ = 0.70

Previous state $h_{t-1}$ = 0.60

Input signal $x_t$ = 0.40

Top: the new state $h_t$ (green) sits on the segment from the past $h_{t-1}$ (blue) to the candidate $\tilde{h}_t$ (orange), at fraction $z$. Bottom: the update gate as a keep-vs-update proportion bar.

Keep the past $(1-z)\,h_{t-1}$ —

Take candidate $z\,\tilde{h}_t$ —

New state $h_t = (1-z)\,h_{t-1} + z\,\tilde{h}_t$ —

Pull $z$ to 0. The new state locks onto $h_{t-1}$ no matter what the input says — this is how a GRU carries a memory across many steps, the same trick as the LSTM's forget gate near 1. Now push $z$ to 1: the past vanishes and $h_t$ becomes the candidate. Separately, drag $r$ to 0 and watch the candidate stop listening to the past entirely — useful when a new phrase should be read fresh.

The Payoff

GRU vs LSTM, side by side

The GRU (Cho et al., 2014) is a streamlined LSTM. It merges the LSTM's separate cell and hidden states into one vector, and folds three gates down to two. Fewer parameters, often comparable accuracy — and the same core idea in both: a multiplicative, near-linear path lets the state travel across time without the $w_h\,\sigma'$ product vanishing or exploding.

	GRU	LSTM
Gates	2 — update $z$, reset $r$	3 — forget, input, output
State	single $h_t$ (cell & hidden merged)	two — cell $c_t$ + hidden $h_t$
Update rule	`h = (1−z)·h_prev + z·h̃`	`c = f·c_prev + i·g; h = o·tanh(c)`
Exposure	whole state exposed each step	output gate filters what is exposed
Parameters	fewer (~3/4 of an LSTM)	more
When	smaller data, faster training, edge/streaming	slightly more capacity on long, complex sequences

One idea, two packagings. Both architectures replace the plain RNN's overwrite with a gated blend. The GRU writes it as a single interpolation, $h_t = (1-z)h_{t-1} + z\tilde{h}_t$; the LSTM splits keeping and writing into separate forget and input gates and adds an output gate. In 2026 both are mostly pedagogical stepping stones to attention and state-space models — but the "gate = learned interpolator" intuition you just dialed in is exactly what those newer models generalize.