Learning outcomes

By the end of this lecture you will be able to:

1. Explain attention as differentiable dictionary lookup.
2. Derive scaled dot-product and justify the √d_k.
3. Distinguish Bahdanau (additive) from Luong (multiplicative).
4. Implement QKV self-attention in PyTorch.
5. Apply causal masking to get a GPT-style decoder.
6. State the O(n²) complexity wall and its consequences.

Recap · where we are

Module 6 opens. The previous lecture ended on a cliff-hanger:

• Seq2Seq works for short sentences.
• The fixed-size context vector can't hold everything for long ones.
• BLEU collapses past ~30 tokens.

Today we fix it.

The one-line fix

Bahdanau et al. 2014 · "Neural Machine Translation by Jointly Learning to Align and Translate."

Why one vector can't hold everything

Think about translating a 40-word English sentence into French. A Seq2Seq encoder has to compress:

• every word's identity
• the full syntactic structure
• the tense, number, gender, mood of every verb and noun
• the coreference chains ("it" refers to "the box")
• the sentiment

...into a single 512-dim vector. One number for every 0.3 bits of meaning.

The shift in viewpoint

Attention is the mechanism for the pull — a differentiable version of "look up the word I need right now."

Four questions

1. What does attention look like — literally, as a heatmap?
2. What are Q, K, V and why "retrieval"?
3. Why do we divide by ?
4. What is self-attention and how does it differ from cross-attention?

What attention looks like

Interpretation

Every target word computes a distribution over source words. Darker cell = more attention mass.

Three things to notice:

1. Rough diagonal — most alignments are monotonic.
2. "traversé" attends to "cross" — semantic alignment across a language barrier.
3. "qu'il" attends to BOTH "it" AND "animal" — coreference resolution, emergent from training.

A 3×3 worked example · before the math

Source · "the cat slept" Target step · decoding French for "cat" → "chat"

The decoder reads a weighted mixture, dominated by the relevant word. That's one step of attention.

Bahdanau (additive) attention · 2014

Score between decoder state and encoder state :

Bahdanau · worked numeric, term-by-term

Toy 2-d. , .

Step 1 · project.

Step 2 · add + non-linearity.

Step 3 · final dot product with .

This single number is the alignment score for one pair. Compute for all , then softmax.

Luong (multiplicative) attention · 2015

A simpler score, no learned MLP:

• Faster — one matrix multiply instead of a two-layer MLP.
• Equivalent in expressiveness when you have enough data (the projections in QKV absorb Bahdanau's ).

This is the version Vaswani et al. kept for the Transformer in 2017, with one small but crucial addition: the scaling.

Why multiplicative won

Two reasons the ML community moved from Bahdanau to Luong:

1. Hardware — a dot product is one matrix multiply; GPUs love it. Bahdanau's MLP has element-wise tanh, which is slower per op and harder to batch.
2. Enough capacity elsewhere — once we added learned projections (next section), we no longer needed the to learn the similarity. The dot product of projections already gives it.

Soft retrieval · picture

Why Q, K, V are three different things · the library analogy

That's it · attention is a soft library lookup. The next slides formalize the math; the analogy is the whole intuition.

The retrieval metaphor

Imagine a Python dictionary lookup:

db = {"cat": "meow", "dog": "bark"}
query = "cat"
result = db[query]      # returns "meow"

Attention is the soft version of this:

• Query — what you're looking for (from the decoder).
• Key — what each encoder state announces itself as.
• Value — what each encoder state actually contains.

Score keys against the query → softmax → use weights to blend values.

Soft retrieval · why three roles not one

You could imagine an attention mechanism where and are the same thing. Early models did exactly this (Luong 2015). So why separate them?

• optimized for similarity with plausible queries.
• optimized to carry whatever downstream layers need.

Separating them doubles the parameter count of one head but roughly doubles the expressiveness too.

The Python-dict mental model · extended

# Hard retrieval
db = {k1: v1, k2: v2, k3: v3}
out = db[query]                      # exact match → one value

# Soft retrieval (attention)
scores  = [sim(query, k) for k in [k1, k2, k3]]   # similarities
weights = softmax(scores)                          # probabilities
out     = weights[0]*v1 + weights[1]*v2 + weights[2]*v3

QKV · the computation

Scaled dot-product · step by step

Scaled dot-product · worked numeric (4 steps)

Tiny example · 2 tokens, .
, , , .

Step 1 · scores.
— token 1's query matches token 2 better (score 2 > 1).

Step 2 · scale.

Step 3 · row-wise softmax.
Row 1:
Row 2:

Step 4 · weighted sum.

Output for token 1 is 33% of + 67% of .

QKV · projection worked example

Input vector for "cat" · (). Project to via three matrices.

The single input now plays three roles: query , key , value .

The full matrices are this calculation repeated for every token. PyTorch:

class AttentionHead(nn.Module):
    def __init__(self, d_in, d_k):
        super().__init__()
        self.Wq = nn.Linear(d_in, d_k, bias=False)
        self.Wk = nn.Linear(d_in, d_k, bias=False)
        self.Wv = nn.Linear(d_in, d_k, bias=False)

    def forward(self, x):
        Q, K, V = self.Wq(x), self.Wk(x), self.Wv(x)
        scores = Q @ K.transpose(-2, -1) / math.sqrt(Q.size(-1))
        weights = scores.softmax(dim=-1)
        return weights @ V

class AttentionHead(nn.Module):
    def __init__(self, d_in, d_k):
        super().__init__()
        self.Wq = nn.Linear(d_in, d_k, bias=False)
        self.Wk = nn.Linear(d_in, d_k, bias=False)
        self.Wv = nn.Linear(d_in, d_k, bias=False)

    def forward(self, x):
        Q, K, V = self.Wq(x), self.Wk(x), self.Wv(x)
        scores = Q @ K.transpose(-2, -1) / math.sqrt(Q.size(-1))
        weights = scores.softmax(dim=-1)
        return weights @ V

Why softmax can get "spiky"

Variance of unscaled dot products

For with i.i.d. entries:

Variance of a sum of independent terms = sum of variances:

(For two independent zero-mean unit-variance variables, .)

Standard deviation → typical magnitudes scale like .

With raw scores are . Softmax of → nearly one-hot. Scale by → variance back to 1.

Worked numeric · the scaling factor in action

. A typical raw-score vector for a query: .

Without scaling.
. Softmax — essentially one-hot → gradient ≈ 0 → learning stalls.

With scaling. Divide by 16: .
, sum . Softmax — soft, gradients flow.

Numeric demo · softmax at different scales

Dot products without scaling behave like the bottom row — effectively one-hot. Divide by and you land back on the top row. The scaling is doing exactly the role of a temperature denominator, derived from variance analysis rather than tuned by hand.

In pictures

Why one-hot is bad

If softmax outputs are (nearly) one-hot, then attention picks one encoder state and ignores the rest.

Two consequences:

1. Information from other positions is discarded.
2. Gradient through the softmax is near zero (saturation) → training stalls.

The fix — divide scores by :

Scores stay in a healthy range → softmax stays soft → gradients keep flowing.

optional · The derivation in three lines

Assume entries are i.i.d. with zero mean and unit variance.

So . Dividing by makes the variance 1 — independent of dimension.

Cross-attention · the decoder reads the encoder

In a Seq2Seq + attention model:

• Queries come from the decoder (current target position).
• Keys and values come from the encoder (all source positions).

This is what the first attention heatmap showed — one distribution per target step over source positions.

Self-attention · the "bank" disambiguation

In "he sat on the river bank" · the word "river" tells you the other meaning.

Self-attention is exactly this · every word looks at every other word in the sentence to figure out its context-specific meaning. The famous "the animal didn't cross the street because it was too tired" example needs this · attention disambiguates "it" by attending to "animal."

Self-attention · a sequence attends to itself

Same operation. Now:

All three come from the same input sequence.

This is the idea that killed RNNs and gave us Transformers (L13).

Self-attention · 3 tokens, by hand

Sentence · "the cat slept" token embeddings .

After projecting: are each a matrix.

Every output row is a weighted blend of value rows — a contextualized embedding for that token.

Self-attention vs convolution · same goal, different bias

Causal mask · 5×5 grid

Causal self-attention · don't peek at the future

scores = Q @ K.transpose(-2, -1) / math.sqrt(d_k)     # (n, n)
mask   = torch.triu(torch.ones_like(scores), diagonal=1).bool()
scores.masked_fill_(mask, float('-inf'))              # future → -inf
weights = scores.softmax(dim=-1)                      # rows still sum to 1

That's the only difference between BERT-style (bidirectional) and GPT-style (causal) attention. Same module, different mask.

Worked numeric · how the −∞ mask works

Raw scores (3 tokens, → no scaling):

Step 1 · mask the upper triangle.

Step 2 · row-wise softmax. Since :

• Row 1:
• Row 2:
• Row 3:

The guarantees future tokens get exact zero weight after softmax. Token 1 sees only itself; token 2 sees 1–2; token 3 sees all.

Complexity · the O(n²) wall

Self-attention on a sequence of length :

• builds an matrix → memory and compute.
• Double the context → 4× the cost.

The four kinds of attention you will meet

Why attention was such a big deal

1. Bottleneck solved · no more "fit everything into 512 dims."
2. Long-range dependencies · every target step can see any source step.
3. Parallelizable · unlike RNNs, all attention scores can be computed at once.
4. Interpretable · attention heatmaps are the first DL visualization tool that's actually informative.
5. Transfer · attention blocks compose cleanly — stack them, mix them with cross-attention, make them multi-headed. The Transformer (L13) is exactly this.

"Attention Is All You Need" · the 2017 pivot

Vaswani et al.'s one-page insight · drop the RNN entirely, use only attention plus FFNs plus positional encodings.

Consequence table:

Every major model since 2018 (BERT, GPT-*, T5, Claude, Llama) is this architecture, plus or minus details. L13 builds it from parts.