Interactive Explainer

A Char-MLP Language Model, Step by Step

Strip a language model down to its bones: a tiny character‑level MLP that reads the last three characters and guesses the next one. Walk one full training example through it — embed, concatenate, hidden layer, softmax, loss, and backprop — one Step at a time, with every number live on the canvas. The finale: the gradient $p - y$ flows back and touches only the embedding rows the example actually used.

~10 min Deep Learning · Language Models · Backprop

Prelude

The whole model on one line

This is makemore in miniature: a vocabulary of $8$ characters, a context of the last $B = 3$ characters, an embedding dimension of $d = 2$, and one hidden layer of width $6$. Every prediction is the same short pipeline:

\underbrace{e_c = E[c]}_{\text{embed}} \;\to\; \underbrace{x = [e_1; e_2; e_3]}_{\text{concat}} \;\to\; \underbrace{h = \tanh(W_1 x + b_1)}_{\text{hidden}} \;\to\; \underbrace{p = \mathrm{softmax}(W_2 h + b_2)}_{\text{next-char probs}}.

The weights below are small fixed numbers (an untrained net), so the run is fully reproducible. Our example: the context is v e e and the true next character is n — the tail of “Na v e e n,” the name Naveen. Step through it.

The Lab

One forward & backward pass

Press Step to advance the pipeline, or jump to any stage with the numbered buttons. The strip below tracks where you are; the panel underneath draws that stage in full.

The current stage, drawn with live numbers.

Model’s top pick —

Prob of true char ‘n’ —

Loss $-\log p_{\text{n}}$ —

Watch stage 6. The gradient on the logits is exactly $p - y$: one negative bar on the true character ‘n’ (push it up) and positive bars everywhere else (push them down). Trace it back and it reaches the embedding table — but only rows v and e glow. The other six rows were never indexed this step, so their gradient is exactly zero.

The Payoff

Next-token prediction, end to end

Every large language model, underneath, is doing what you just stepped through — only wider, deeper, and with attention in place of a single concatenation. The skeleton is unchanged:

Embed each context token into a vector by indexing a table $E$.
Mix the context (here: concatenate, then one $\tanh$ layer) into a hidden state.
Score & normalise with a linear layer plus softmax to get next‑token probabilities.
Grade with cross‑entropy $-\log p_{\text{true}}$, and backprop the clean signal $p - y$.

Sparse updates. Because embeddings are a lookup, one example’s gradient only edits the rows it used. Over a whole corpus, characters (or words) that share contexts get nudged in similar directions and drift together — which is exactly why the embedding geometry ends up meaningful. Embedding → MLP → softmax, graded by $p - y$: that is the entire idea of learning to predict the next token.