Interactive Explainer

Generate, Token by Token

A language model does not write a sentence — it predicts one next token, samples it, sticks it on the end, and does the whole thing again. Press Step to run one round of that loop over a tiny vocabulary, and turn the temperature, top-k and top-p knobs to watch the next-token distribution sharpen, narrow, and change what gets sampled.

~9 min Deep Learning · Transformers · Decoding

Prelude

Autoregression is a loop

Give the model a prefix of tokens $x_{1:t}$. It runs one forward pass — embed, causal self-attention, feed-forward — and emits a vector of logits, one score per vocabulary word. A softmax turns those into a probability for the next token:

p(x_{t+1}\mid x_{1:t}) = \operatorname{softmax}\!\big(z / T\big), \qquad x_{t+1} \sim p .

Then the sampled token is appended and the loop repeats with $x_{1:t+1}$. Nothing about the model changes between steps — only the context grows. The interesting choices all live in that last arrow, $x_{t+1}\sim p$: how we turn a distribution into a single word. Temperature $T$, top-k and top-p (nucleus) are three knobs that reshape $p$ before we draw from it.

The Lab

Run the decoding loop

Pick a prefix, set the knobs, and press Step. Each press shows the next-token distribution over a fixed 12-word vocabulary, samples one token (highlighted), and appends it. The logits are fixed and seeded — a small stand-in model — so the distribution is fully determined by the context and your knob settings.

Prefix:

prefix x₁:ₜ → embed → causal self-attention → FFN → logits z → sample xₜ₊₁

Temperature $T$ = 1.00

Top-k off

Top-p (nucleus) $p$ = 1.00

Next-token distribution over the vocabulary. Dashed outline = after temperature only; solid bar = after top-k / top-p (renormalized). Grey = filtered out ($p=0$). The arrow marks the token that was sampled.

after temperature kept (renormalized) filtered out sampled token

Last sampled —

Candidates kept —

Entropy (bits) —

Top prob —

Turn temperature to 0.1. The distribution collapses onto its single most likely token — near-greedy, repetitive, safe. Push it to 2.0 and it flattens toward uniform — surprising, sometimes incoherent. Now set $T=1$ and instead pull top-p down to 0.6: the long tail of unlikely words greys out entirely, so you keep the fluency of a warm temperature without ever sampling something absurd. That is why nucleus sampling is the modern default.

The Payoff

Three knobs, one trade-off

Every decoding knob is negotiating the same tension: diversity (explore more of the distribution) against focus (stay on the high-probability words the model is confident about). They just cut the distribution differently.

Temperature rescales the logits before softmax. $T<1$ sharpens toward the top token; $T>1$ flattens toward uniform. It reshapes every probability but removes nothing.
Top-k keeps only the $k$ highest-probability tokens and zeroes the rest — a hard cap on how many candidates survive, regardless of how the mass is spread.
Top-p (nucleus) keeps the smallest set of tokens whose probabilities sum to at least $p$. The cut adapts to the shape: a confident step keeps one or two words, an uncertain step keeps many.

Why sampling at all? Always taking the arg-max (greedy) is deterministic and often degenerates into loops. Sampling injects the controlled randomness that makes generations varied and human-sounding — and top-k / top-p are the guardrails that stop that randomness from wandering into the model's low-probability nonsense. Decoding is where a fixed model becomes a writer with a personality you can dial.