Interactive Explainer

Inside a Transformer Block

A transformer is a stack of identical blocks, and every block does the same six little things to a running residual stream of token vectors. Press Step to walk one block end to end: normalize, let attention mix information between positions, add it back to the stream, normalize again, let the feed-forward network transform each position on its own, and add that back too.

~10 min Deep Learning · Transformers · Self-Attention

Prelude

One block, six moves, one stream

Feed a short sequence of tokens in as rows of a matrix $X \in \mathbb{R}^{n\times d}$ — $n$ positions, each a $d$-dimensional vector. A transformer block does not replace $X$; it edits it. Two sublayers each read the stream, compute a correction, and add that correction back. This is the residual stream, and its two updates are:

X \;\leftarrow\; X + \operatorname{Attention}\!\big(\operatorname{LN}(X)\big), \qquad X \;\leftarrow\; X + \operatorname{FFN}\!\big(\operatorname{LN}(X)\big).

Everything hinges on who talks to whom. Attention is the only place information moves between positions; the feed-forward network touches each position independently. LayerNorm and the residual adds are the plumbing that keeps this deep stack numerically calm. The widget below takes four tokens through exactly one block, one move at a time.

The Lab

Step the residual stream through one block

The diagram is the wiring; the heatmaps are the actual numbers (fixed, seeded). Warm cells are positive, blue cells negative, intensity is magnitude. Press Step to perform the next of the six moves — watch the residual stream (bottom) change only at the two “+” adds, never during the branch computations.

The residual stream (green when it updates) runs straight down; each sublayer branches off, is computed, and is added back at a “+”.

Start The four token vectors sit in the residual stream, untouched. Press Step to begin the block.

Current move —

Information flow —

Stream norm $\lVert X\rVert$ —

The current move, computed on a branch off the stream.

The residual stream $X$ (nx d). It changes only at the two residual adds; LayerNorm, attention and the FFN all read it but write to a branch first.

negative ~zero positive tokens: The · cat · sat · down

Watch the attention move. At step 2 the $n\times n$ weight matrix lights up: row $i$ tells you how much position $i$ pulls from every position $j$. The output for “down” is literally a blend $\sum_j A_{ij}\,v_j$ of all four value vectors — that is information travelling between tokens. Nowhere else in the block does that happen.

The Payoff

Why this shape, and why it stacks

A block is a careful division of labour. Attention decides what to read from where; the FFN decides what to do with what each position now holds. Separating “move information” from “process information” is what lets the same block be stacked dozens of times.

Attention mixes positions. Every output row is a weighted average of value vectors from all positions. This is the one and only channel for tokens to influence each other.
The FFN is position-wise. The same little MLP is applied to each row on its own — no row ever sees another. It is where most of a transformer's parameters (and much of its stored knowledge) live.
Residual + norm keep the stream stable. Because each sublayer only adds a correction, gradients get a clean highway straight back to the input, and LayerNorm keeps every row at a sane scale so 96-layer stacks still train.

Pre-norm vs post-norm. This widget shows the modern pre-norm arrangement — LayerNorm sits inside each branch, so the residual stream itself is never normalized, only added to. That untouched highway is exactly why very deep transformers (GPT, LLaMA) train stably where the original post-norm design needed careful warm-up.