Interactive Explainer

Embeddings are a Lookup Table

Networks cannot multiply by the letter “e.” So the very first layer of a language model keeps a table — one learnable row of numbers per token — and “embedding a token” is nothing more fancy than reading off its row. Pick a token, watch its row light up, and drag it around a 2‑D map to feel what training is really doing: pulling related tokens together.

~7 min Deep Learning · Language Models · Representations

Prelude

A row per token

A token id is just an integer — token 3 is no bigger than token 1, they are only names. To turn that name into something a network can compute with, we store a matrix $E \in \mathbb{R}^{V \times d}$: one row for each of the $V$ tokens, each row a vector of length $d$. Embedding token $i$ is a single indexing op:

e = E[i] \in \mathbb{R}^{d}, \qquad \text{(in code:}\;\; \texttt{e = E[i]}\text{)}.

No multiplication, no dot product — just a lookup. (It is equivalent to multiplying a one‑hot vector by $E$, which is why it is a “layer,” but the one‑hot times matrix collapses to picking one row.) Below, $V = 10$ tokens and $d = 2$, so every token is both a row of the table and a point on a map.

The Lab

Look it up, then move it

Click a token. Its row in the table (left) lights up and is read out as a dense vector; the same token is ringed on the 2‑D map (right). Then nudge its two coordinates with the sliders — or just drag the point — and watch its row and its position move together. They are the same numbers.

Token id

The embedding table $E$ (10 rows × 2 dims). The selected token’s row is its vector $e = E[i]$.

Every token as a point $(e_0, e_1)$. Drag the ringed point; the dashed line shows its nearest neighbour.

Dim 0 $e_0$ = 0.00

Dim 1 $e_1$ = 0.00

Lookup e = E[i] —

Selected token —

Nearest neighbour —

Distance to it —

Try this. The five vowels start clustered in one corner and the five consonants in another — the kind of structure a trained model discovers on its own. Now drag a vowel down into the consonants: its nearest neighbour flips to a consonant. Nothing about the token “changed” — only its row did. Similarity in an embedding space is entirely a matter of which rows sit near which.

The Payoff

Training is just moving rows

Because embedding a token is pure indexing, the only thing gradient descent can do to the embedding layer is edit the rows that were used. When two tokens keep appearing in similar contexts, the loss keeps nudging their rows in similar directions, and they drift together — exactly the clustering you just dragged apart by hand.

A token id carries no meaning; the meaning lives in the row the model learned for it.
“Distance between tokens” is just distance between rows — there is nothing else.
A step of training touches only the rows of the tokens in the batch; every other row is untouched. (That is the star of the companion char‑MLP explainer.)

One layer, two views. The heat‑map and the scatter plot are never out of sync because they render the same array. An embedding matrix is simultaneously a table you index into and a geometry you can measure — and that geometry is the model’s entire idea of what these tokens mean.