Attention, Calculated

Prelude

The ambiguity problem

A word in isolation is almost meaningless. Consider the word bank. Is it a river bank or a money bank? The word alone can't tell you—you have to look at its neighbors.

Self-attention is how Transformers let each word ask around. Every word in a sentence says to every other word: "Hey, how much should I care about you?" The answer comes back as a single number between 0 and 1. Then each word rebuilds itself as a weighted mix of its neighbors' meanings.

This page makes all of that explicit. By the end you'll have pushed raw scores through a softmax, watched a blended value vector form, and built the intuition for why Transformers beat every prior architecture at disambiguation.

Pick your sentence

We'll work through a handful of classic ambiguous words. Pick one and the whole page follows it:

The focus word bank needs to figure out whether it is water-related or money-related by listening to its neighbors.

Step 1

Three roles for every word

In a Transformer, every word is associated with three separate vectors, each playing a different role:

Query ($Q$): "What am I looking for?" This is the perspective of the word that wants to update itself.
Key ($K$): "What do I offer?" Every other word advertises its flavor here.
Value ($V$): "If you decide to attend to me, here is the actual meaning I contribute."

Queries and keys determine how much to attend. Values determine what gets mixed in. Keeping those two roles apart is what makes attention expressive.

The library analogy. Imagine walking into a library with a question written on a sticky note (your query). Every book has a spine label (key) and a bunch of text inside (value). You match your sticky note to the spine labels, pull out the books that fit, and read their text. That's one step of attention.

Step 2

Where Q, K, V actually come from

Step 1 painted Q, K, V as three roles a word plays. Where do those vectors physically come from? Each is a linear projection of the same input embedding $x_i \in \mathbb{R}^{d_{\text{model}}}$. Three matrices $W_Q, W_K, W_V$ are learned once, shared across every position:

Same input, three different rotations. The query rotation pulls out the "what am I looking for?" features, the key rotation pulls out the "what do I advertise?" features, and the value rotation pulls out "what do I deliver if attended to?". One word plays three roles because three separate matrices were trained to read it three different ways.

Worked example, real numbers

Click any token to see its raw embedding $x_i$ pushed through $W_Q$, $W_K$, $W_V$. We use $d_{\text{model}}=4$ and $d_k = d_v = 2$ so the matrices fit on screen — in GPT-style models they're 768 or 4096, but the recipe is identical. (Embeddings and projection matrices are pre-cooked here; in real training they'd be learned by gradient descent — see Step 8.)

Embedding $x_i$

$1 \times 4$

×

$W_Q$

$4 \times 2$

$W_K$

$4 \times 2$

$W_V$

$4 \times 2$

=

$q_i$

$1 \times 2$

$k_i$

$1 \times 2$

$v_i$

$1 \times 2$

Pick a token to see its projection.

Why three matrices and not one? A single rotation can express one feature view at a time. With three independent matrices, the model can simultaneously make a word "ask for verbs", "advertise as a water-related noun", and "deliver geography content" — three orthogonal jobs in three orthogonal subspaces.

Step 3

Dot product = compatibility score

How do we measure how well a Query matches a Key? We use the dot product. If the two vectors point the same way, the score is large and positive. If they're perpendicular, zero. If they point opposite ways, negative.

Drag the orange Query arrow for the focus word, and the blue Key arrows for every other word. Watch the raw scores update live below. Drag the query so it aligns with one specific key and you'll see that word win.

2-D stand-in for the real high-dimensional Q/K space. All arrows are constrained to the unit circle.

Word	Query vector	Key vector	Dot product

Scaled dot product in real Transformers. In production, the dot product is divided by $\sqrt{d_k}$ (where $d_k$ is the key dimension). With large dimensions, raw dot products balloon and squash softmax to one-hot. The scale factor keeps the gradient healthy. We ignore it in 2-D because the effect is tiny.

Step 4

Softmax turns scores into percentages

Raw dot products aren't weights—they can be negative, or gigantic. We need them to behave like probabilities that sum to one. Softmax does exactly that:

Exponentiating amplifies differences: a score that's just a bit higher than the others rockets upward, dominating the weights. This is why softmax is called "soft" argmax—it picks a winner, but still passes non-zero signal to everyone else.

Step-by-step calculation (live)

Here's the full pipeline applied to your current dragged arrangement. Every number comes from the canvas above:

Word	Raw score	exp(score)	Weight (softmax)

Notice the winner-take-most behavior. Make one of your dot products visibly larger than the rest. You'll see its softmax weight jump above 80% while the others shrink to single digits. Now nudge another key vector toward the query— the new word steals a huge chunk of attention even for a tiny tilt. Small changes in scores mean large changes in weights.

Step 5

Blend the values into a new meaning

Now the payoff. The focus word rebuilds its meaning as a weighted sum of the Value vectors, with the softmax weights we just computed:

Below, the teal arrows are the value vectors for every word (drag them to simulate different "meanings"). The dashed orange arrow is $V_{\text{final}}$—the new contextual meaning for the focus word. Watch it slide toward whichever value has the highest attention weight.

Blend of value vectors by the current attention weights. Dashed orange: the output $V_{\text{final}}$.

Why keys and values are kept separate. At first glance you might ask: why not use the value as both the key and the thing to be averaged? Because then you couldn't express "attend to the word river because it's a water cue, but grab its broader geography meaning, not the raw token." Keys ask "does this match?"; values answer "here's what to contribute if I do."

Step 6

All at once — the matrix view

Real Transformers don't loop over tokens one at a time. They stack all $N$ queries into a matrix $Q$, all $N$ keys into $K$, all $N$ values into $V$, and produce the entire layer output in a single matrix multiplication:

Here's that exact computation, run live on your current sentence. The score matrix $S$ is the table of every query against every key. Apply softmax to each row and you get the attention weight matrix $A$. Multiply by $V$ and you get the new embeddings $Y$ — one new vector per token, all in one matmul.

Score $S = QK^\top / \sqrt{d_k}$

$N \times N$

softmax
(row-wise)

Weights $A = \mathrm{softmax}(S)$

$N \times N$

·V

Output $Y = AV$

$N \times d_v$

Inspect row: Pick a row to see one token's outgoing attention.

Read it like a table. Row $i$ of $A$ is "how token $i$ distributes its attention". It must sum to 1 across the row. Column $j$ tells the opposite story: "who is paying attention to token $j$?". The diagonal $A_{ii}$ is self-attention — how much each token listens to itself.

Step 7

Three knobs that shape every Transformer

The clean formula above hides three design choices that decide whether the layer is useful or broken. Here they are with quick interactives.

Knob 1 — Why divide by $\sqrt{d_k}$?

Dot products of two random $d_k$-dimensional vectors have variance $d_k$. As $d_k$ grows, raw scores get huge, softmax sharpens to one-hot, and gradients vanish. Dividing by $\sqrt{d_k}$ keeps the variance at 1 regardless of width.

$d_k$ 8

divide by $\sqrt{d_k}$

Sample of softmax weights for one query against 8 random keys

—

Histogram of max-weight across many random queries

—

Knob 2 — Causal masking

For a language model, the word at position $t$ must not peek at positions $t+1, t+2, \dots$ — or training would just teach it to copy the next token. We zero out those entries before the softmax by adding $-\infty$ to the upper triangle. After softmax, the future is exactly 0.

Unmasked weights $A$

+ mask

Causal $A_{\text{causal}}$

Each row's surviving weights are renormalised by the softmax, so an early token only sees itself, the next sees itself plus one predecessor, and so on. This single trick is what makes GPT-style models autoregressive.

Knob 3 — Multiple heads

One attention map can only express one relation at a time. Real Transformers run $h$ heads in parallel — each with its own $W_Q^{(h)}, W_K^{(h)}, W_V^{(h)}$ — then concatenate their outputs. Different heads learn to specialise (syntax, coreference, positional adjacency, topic).

Each head sees the same input but uses a different rotation of it (we simulate that here with four hand-picked rotations). Real heads are learned, but the visual story is the same: they notice different things.

Step 8

Positions: the bug attention secretly has

Self-attention is permutation-invariant. Shuffle the order of the input tokens and the output is the same set of new vectors, just reshuffled. For language that is fatal — "dog bites man" and "man bites dog" should not be the same.

Quick proof. Reorder the tokens in $X$ by some permutation $P$. Then $Q, K, V$ also permute: $Q' = PQ$, $K' = PK$, $V' = PV$. The new attention output is $\mathrm{softmax}(PQ\,(PK)^\top/\sqrt{d_k})\,PV = P\,\mathrm{softmax}(QK^\top/\sqrt{d_k})\,V$. The set of output vectors is identical; only their order changes.

The fix — add a position signal

Before any attention runs, the model adds a position vector $p_t \in \mathbb{R}^{d_{\text{model}}}$ to each token's embedding. The classic Vaswani et al. choice is sinusoidal:

Each dimension is a sinusoid with a different wavelength. The short-wavelength dimensions distinguish nearby positions; the long-wavelength ones carry far-apart structure. Together they give every position a unique fingerprint — and crucially, a fingerprint where relative position can be linearly decoded.

Sinusoidal positional encodings. Rows are positions, columns are dimensions. Shorter wavelengths on the left, longer on the right. Hover-pick a row in the controls below to see one position's full vector.

positions $T$ 32

dim $d$ 32

highlight position 5

With vs without — permute the sentence

Toggle the position signal on and off, then shuffle the words. With positions, "river bank" and "bank river" produce different attention; without, they're identical.

add positional encoding

Token order

Attention weights for focus token

Modern variants. The original sinusoidal encoding is simple and extrapolates to longer sequences. Modern systems often use learned embeddings (BERT), rotary embeddings (RoPE, used in LLaMA & DeepSeek — rotates Q and K instead of adding to $X$), or ALiBi (a fixed bias added straight to the score matrix). All solve the same problem: telling attention where each token is.

Step 9

Watch a tiny attention head learn

Every Q, K, V in the article so far was hand-tuned to produce the "right" answer. Real Transformers don't get those for free — they discover them with gradient descent. Let's train one in your browser, end-to-end, with manually-coded backprop. Every parameter, every gradient, every step is in plain JavaScript — nothing imported.

The toy task — a soft lookup

We give the model a sequence of four tokens. The first three are candidates: each carries a one-hot identity in $\{A, B, C\}$ plus a random 3-dim payload. The fourth is the query: a one-hot saying which candidate's payload we want, with zeros for its payload slot. The model has to output that payload at the query position — a real "look-up by name" problem.

The only learnable parameters are the three projection matrices $W_Q$, $W_K$, $W_V$. We initialise them to small random Gaussians, run gradient descent on mean-squared error at the query position, and watch the loss fall.

learning rate 0.10 step 0 loss —

Loss curve (log scale)

Press Start training to begin.

Attention weights on a fixed example

—

What the model is learning

$W_Q$

$W_K$

$W_V$

What to watch for. Within roughly 100 steps the loss drops by an order of magnitude. The query row of the attention map snaps to (almost) one-hot at the matching candidate. $W_Q$ and $W_K$ start random, but quickly grow large entries on the id columns (so the dot product picks up matching ids); $W_V$ grows large entries on the value columns (so the right payload gets carried through). Nobody told the model "look up by id" — it discovered that strategy because it's the cheapest way to drive the loss down.

Step 9 ½

Two heads, two specialists

Step 7 said multi-head exists; Step 9 trained one head. Now watch two heads train side-by-side on a task that a single head cannot solve. Each head gets its own $W_Q, W_K, W_V$ (and a final linear that combines the two outputs). Loss flows through both.

The task: two compatible lookups

The sequence has four tokens: two type-A pairs $(\text{key}_A, \text{val}_A)$ and two type-B pairs $(\text{key}_B, \text{val}_B)$. Each token's input is $\bigl[\text{type}\ (2)\;|\;\text{key}\ (3)\;|\;\text{val}\ (3)\bigr]$. The query token (token 4) carries $\bigl[\text{type-A flag}\ (1)\;|\;\text{key}\ (3)\;|\;\text{type-B flag}\ (1)\;|\;\text{key}\ (3)\bigr]$ and we ask the model to output $\text{val}_A + \text{val}_B$ at the query position. One head can't disentangle the two lookups; two heads can.

learning rate 0.05 ablate to single head step 0 loss —

Head A attention (on a fixed example)

Head B attention (on the same example)

Loss curve

Head specialisation score

How peaked each head is on its preferred type. Specialisation rises after ~80 steps once the heads carve up the task.

What to watch. With two heads the loss falls smoothly to near zero; one head's heatmap focuses on the type-A pair, the other on the type-B pair. Tick ablate to single head and the loss plateaus much higher — one attention pattern can't point at two different things at once. This is the actual reason real Transformers run 8 or 16 heads.

Step 10

Run it on all five sentences

Here's one more look at the pipeline, this time fully automated. The table below evaluates the same Q, K, V vectors (the pre-set "sensible" ones for each ambiguous word) through the entire attention formula and shows which neighbor dominates. Click a different sentence to see a different winner.

Sentence	Focus word	Top-attended neighbor	Attention %	Interpretation

Observe. In every one of the five sentences, the ambiguous word's query lines up with exactly one neighbor that disambiguates it. "Bank" listens to "river" in one case, to "money" in another. "Apple" tunes in to "pie" versus "phone". That's disambiguation by attention—no hand-coded rules, just vector alignment.

Step 11

Three things attention is not

Myth

"Attention is just a weighted average, nothing new."
The weights themselves are computed from the data through $QK^\top$—they depend on every pair of tokens. No static weighted sum gives you that. It's a function whose coefficients are learned on the fly.

Myth

"Attention tells you what the model cares about."
Attention weights are one signal inside a deep model. Cutting a low-weight neighbor doesn't necessarily change the prediction, and the same logit can be reached via many weight patterns. Reading them as explanations is risky.

Myth

"Every word attends to every other word equally in cost."
Self-attention is $O(n^2)$ in sequence length: doubling the sentence quadruples the compute. That's why long-context models use tricks (sparse, linear, sliding-window) to cut this down.

Step 12

What we still swept under the rug

You've now seen every number of one head, one layer, with positions, with masking, with multi-head, and with training. A real Transformer keeps going:

Many stacked layers. The output of one attention block becomes the input to the next. After a few layers, "bank" carries meaning shaped by chains of inference across the whole sentence. ViT-Base stacks 12, GPT-3 stacks 96, GPT-4 reportedly more.
Residual streams + LayerNorm. Each block is wrapped in $x \leftarrow x + \mathrm{Attn}(x)$ followed by a LayerNorm and an MLP. The residual lets information skip unchanged through layers and is now believed to be the actual "memory tape" the model writes onto.
Cross-attention. In encoder-decoder models (translation, T5, image-to-text), $Q$ comes from the decoder but $K, V$ come from the encoder — the decoder "asks the source" instead of itself. Mechanically the same formula.
Efficient attention. $O(N^2)$ blows up at long context. FlashAttention reorders the matmul to fit in SRAM; sliding-window (Mistral), sparse (Longformer), state-space (Mamba), and linear-attention (Linformer) variants trade a little expressivity for $O(N)$ or $O(N \log N)$.
KV cache for inference. When a chatbot generates text token by token, the keys and values of all previous tokens are cached and reused. That cache is most of what fills GPU memory in production deployments.

Final takeaway. Self-attention is a Transformer's universal way of asking "which neighbors should I listen to?" You just computed every step of that answer by hand — from the projection matrices that produce Q, K, V, through the matrix-form softmax, through positional fixes, through gradient descent on a real toy. Stacking this operation in depth and width, with residuals, MLPs, and a billion-token training set, is what builds GPT-level understanding from raw tokens.