Interactive Explainer

LSTM Gates Playground

An LSTM cell has three gates: a forget gate decides what to erase from the cell state, an input gate decides what to write, and an output gate decides what to read. Drag each slider, feed an input sequence, and watch the cell state flow — or freeze in place.

~8 minDeep Learning · RNN · LSTM

Vanilla RNNs struggle with long-range dependencies. Gradients multiplied through a chain of W^⊤·diag(tanh') either vanish or explode. Hochreiter & Schmidhuber's 1997 insight: give the recurrence its own state that is updated by gates — sigmoid values in [0, 1] — instead of a matrix multiply.

The cell state becomes a conveyor belt. At each timestep, the forget gate multiplies, the input gate adds, and only the output gate decides how much to expose as the hidden state.

The playground

forget (f) 0.90 input (i) 0.30 output (o) 1.00

        Input spikes at t=3 and t=7 · solid line = cell state c_t · dashed = hidden state h_t
      

Try these presets

Remember forever — f=1, i=0. The cell state freezes after the first input. This is how LSTMs hold long-range information.
Forget always — f=0. Every timestep erases. Like vanilla RNN with tanh saturation.
Overwrite always — f=0, i=1. Cell state tracks the latest input only.

The key equation. c_t = f_t · c_{t-1} + i_t · c̃_t. The forget gate is a multiplicative path (no matrix multiply), so gradients through the cell state don't explode or vanish — they scale by f_t, which the network learns to keep near 1 when memory matters.

LSTM vs GRU

GRU (Cho 2014) merges the forget and input gates into one "update" gate, and merges c and h. Fewer parameters, often comparable accuracy. Picking between them in 2026 is mostly personal preference; both are dwarfed by Transformers on most tasks. But for tiny devices and streaming, they are still the workhorses.

BPTT and the vanishing-gradient story

Training an RNN is backprop-through-time (BPTT): unroll the recurrence, treat it like a feed-forward net with shared weights, and propagate gradients backward through time. The fatal problem in a plain RNN: at every step backward, the gradient is multiplied by the same Jacobian of the recurrent map. If that Jacobian's spectral radius is < 1, gradients vanish exponentially; if > 1, they explode.

∂L/∂h_0 = ∂L/∂h_T · ∏_{t=1}^{T} ∂h_t/∂h_{t−1}

Why LSTMs fix it: the cell state $c_t$ carries information forward via $c_t = f_t \odot c_{t-1} + i_t \odot g_t$. The Jacobian $\partial c_t / \partial c_{t-1} = f_t$ is just the forget-gate value — a per-step learnable scalar in $(0,1)$. When the model wants to remember, $f_t \approx 1$ and the gradient flows back unchanged. That's the actual reason LSTMs train.

Peephole / ConvLSTM / xLSTM

Peephole LSTM. Gates also see the cell state $c_{t-1}$. Marginal accuracy on long-dependency tasks.
GRU. One fewer gate; one fewer state; comparable on most tasks; cheaper.
ConvLSTM. Replace the matmuls with convolutions; useful for spatio-temporal data (radar, video, satellite time series).
xLSTM (2024). Updated gates + matrix-valued memory; scales further; competitive with small Mambas on language modelling.

When to reach for an LSTM in 2026

Streaming inference where state-space carrying matters per step (low-latency speech, time-series forecasting at the edge).
Tiny embedded devices where Transformers' KV cache is too expensive.
As a baseline before reaching for Mamba or PatchTST on long-horizon time series.

Beyond that, modern Transformers and SSMs dominate. The LSTM lives on as the cleanest pedagogical model for the vanishing-gradient problem and the simplest model where "memory" is an explicit, gated state.

Companion: Mamba / SSMs · vanishing gradients · PatchTST forecasting.