Learning outcomes

By the end of this lecture you will be able to:

1. Describe encoder-decoder RNN architecture for variable-length tasks.
2. Implement teacher forcing and explain exposure bias.
3. Contrast greedy / beam / top-k / nucleus decoding.
4. Apply length normalization in beam search.
5. Diagnose the fixed-length bottleneck that killed pre-attention Seq2Seq.
6. Motivate attention (L12) as the fix for the bottleneck.

Recap · where we are

Last lecture: LSTMs solve the vanishing-gradient problem in RNNs via gated cell states.

But solving depth is not enough. Many tasks need to map an input sequence to a different output sequence:

• Machine translation — English to French
• Summarization — article to abstract
• Speech recognition — audio to text
• Program synthesis — comment to code

Four questions

1. How do we map a variable-length input to a variable-length output?
2. What is teacher forcing and why does everyone use it?
3. How do we decode at inference — greedy, beam, nucleus?
4. What's wrong with Seq2Seq — and why did attention become inevitable?

Seq2Seq · the 2014 breakthrough

Two separate RNNs:

1. Encoder · reads the source sequence , updates its hidden state.
2. Context vector · encoder's final hidden state — a compressed summary of the whole source.
3. Decoder · starts from , generates target tokens one at a time.

The whole idea in one sentence

Compress source into a vector · decompress into target.

The same encoder-decoder pattern returns in T5 (L14), Stable Diffusion (L22), and every modular ML system that maps between domains.

Shared vs separate vocabularies

Two design choices:

• Separate vocab · source is 40k English tokens, target is 40k French tokens, each with its own embedding matrix. Clean; embeddings can specialize.
• Shared vocab · one vocabulary for both, one embedding matrix. Saves parameters; lets the model see "Paris" as the same token in both languages.

The translator analogy

The problem · for long sentences, even a great human's mental summary fails. So does Seq2Seq · this is why we'll add attention in L12.

The big picture · encoder–decoder

The architecture

A quick look inside nn.LSTM

An nn.LSTM layer is a function with specific I/O:

• Input · sequence of embedded tokens (e.g. 10-word sentence → (10, 256)).
• Output · returns two things:
  1. outputs — hidden state at every time step (e.g. (10, 512)).
  2. (h_n, c_n) — final hidden + cell state (each (1, 512)).

For the encoder we want the final state — that's our context vector . We discard outputs (using _ in Python) and keep the tuple (h, c).

Seq2Seq in PyTorch · skeleton

class Seq2Seq(nn.Module):
    def __init__(self, src_vocab, tgt_vocab, d_emb=256, d_h=512):
        super().__init__()
        self.src_emb = nn.Embedding(src_vocab, d_emb)
        self.tgt_emb = nn.Embedding(tgt_vocab, d_emb)
        self.encoder = nn.LSTM(d_emb, d_h, batch_first=True)
        self.decoder = nn.LSTM(d_emb, d_h, batch_first=True)
        self.output  = nn.Linear(d_h, tgt_vocab)

    def forward(self, src, tgt):
        _, (h, c) = self.encoder(self.src_emb(src))     # context = (h, c)
        dec_out, _ = self.decoder(self.tgt_emb(tgt), (h, c))
        return self.output(dec_out)

The problem with training auto-regressively

At inference the decoder feeds its own predictions back in:

decoder sees "<start>" → emits "The" → sees "The" → emits "animal" → …

But at training, if the decoder's first prediction is wrong, the error compounds — all subsequent steps condition on bad inputs. Training becomes painfully slow and unstable.

Teacher forcing · the fix

Teacher forcing · detailed flow

Teacher forcing · the training-wheels analogy

Two regimes side-by-side

Why teacher forcing? · speed

The biggest reason for teacher forcing isn't safety — it's parallelism.

• Autoregressive · compute step 1 → step 2 → step 3 → … sequential.
• Teacher forcing · all decoder inputs <s>, "The", "cat", … are known up-front → feed them in all at once → one big matrix multiply.

A slow sequential loop becomes a fast parallel computation — 10–100× speedup on training. Empirically tolerated despite the "you train on a distribution you won't see at inference" critique.

Teacher forcing · why it's OK pragmatically

Even though training ≠ inference, teacher forcing works because:

1. Loss is averaged over all time steps · bad predictions at step 10 aren't penalized harder than bad predictions at step 1. Every position gets balanced gradient.
2. The model learns conditional distributions · if you give it the right at inference (as in a perfect rollout), it generalizes.
3. Huge speedup from parallelization · with ground truth, all decoder steps can be computed in parallel (one big matrix multiply) instead of sequentially.

Exposure bias · the price you pay

Exposure bias · concrete cascade

Source: "Le chien a chassé le chat." Target: "The dog chased the cat."

Training (teacher forcing).

1. Input <s> → predicts "A" (logP −1.2). Truth is "The". Loss is computed.
2. Next input is the ground truth "The". Back on track.

Inference (autoregressive).

1. Input <s> → predicts "A".
2. Next input is "A" (its own mistake).
3. Conditioned on "A", the model now predicts "dog" (probable continuation of "A").
4. Sequence so far · "A dog…". The model has never seen its own mistakes during training, so it has no idea how to recover. Continues "A dog ran away" — completely diverged.

The model trained in a perfect world, tested in a messy one.

Mitigations · scheduled sampling (Bengio 2015), noisy data augmentation, or just sidestep with massive-scale Transformers (2020+).

Four common strategies

Decoding · the search tree

Greedy · pick top-1 every step

Simplest: at each step, pick .

Fast. Deterministic. Usually suboptimal — a slightly-less-likely next token can lead to a much-more-likely full sequence.

Example (simplified):

greedy: "The dog is running" (but doesn't quite fit context)
better: "The puppies are running" (total prob higher)

Greedy fails · the canonical example

Imagine the true best translation is "The cat sits on the mat" (probability 0.7).

At step 1, probabilities are:

• "The" · 0.6 (leads to the correct sequence)
• "A" · 0.8 (leads to "A feline rests...", probability 0.5)

Greedy picks "A" because it's locally higher. But the full-sequence score is lower than starting with "The".

Beam search · the tree

Beam search · the team-of-hikers analogy

The team's collective best end-of-path score is far closer to the global maximum than greedy's. Cost · compute. Quality · 2-5 BLEU points typically.

From probabilities to log-probabilities

We want the sentence with highest :

For a 20-word sentence we'd multiply 20 small probabilities → numerical underflow (rounded to 0).

Fix · take logs. . Multiplying probabilities becomes adding log-probabilities — much more stable.

Each is a negative number. Higher (less negative) = better.

The length-bias problem

Every is negative → longer sentences accumulate more negatives → lower scores → always preferred shorter.

Plain log-prob picks "I am". Wrong.

Length normalization · divide by sentence length raised to :

• · no penalty (broken).
• · simple average per token.
• – · typical compromise (a soft penalty).

It makes finished, longer hypotheses comparable to short ones — the goal is fair comparison, not always preferring long.

Beam search · keep top- paths

At each step, maintain candidate prefixes. Expand each by all next tokens, keep the top by length-normalized log-prob score.

Typical to . More = better quality, slower. Still deterministic given .

Beam search · worked example with

Vocab · {The, A, cat, dog, sat, ran}. Decoding "The cat sat".

A beam of already finds "The cat sat" (logP -1.7) where greedy might have committed early to "A" and ended at "A dog ran" (logP -2.5).

Top- and nucleus · the improv-comedian analogy

Top- vs nucleus · how the pool is chosen

For open-ended generation (story writing, chat) beam is too deterministic — everything sounds the same.

• Top- · pool = the most-likely tokens. Fixed size. Renormalize, sample.
• Top- (nucleus) · pool = smallest set with cumulative prob . Dynamic size — small when the model is confident, large when uncertain.

When the next-token distribution has one tall bar, nucleus narrows automatically; when it's flat, nucleus widens. Top- ignores this and uses the same everywhere.

The failure mode · source length

The bottleneck in one sentence

For short sentences, fine. For long sentences, the encoder forgets the beginning by the time it reaches the end. The decoder has no way to recover what was lost.

Sutskever's own fix · reverse the input

The 2014 paper itself found that reversing the source improved BLEU by several points:

"I am happy" → encoded in order
"happy am I" → encoded reversed

Why? The last source words (now first) are closest to where the decoder begins generating — less path length for that information to travel through the hidden-state chain.

The obvious next step

If one context vector can't hold all source info, don't use one context vector.

Instead, let the decoder look at all the encoder hidden states — and decide which ones to focus on for each target step.

Where Seq2Seq-like ideas still ship

• Speech recognition · Whisper uses encoder-decoder with attention (the full recipe).
• Machine translation · Transformer encoder-decoder is literally Seq2Seq with attention instead of a context vector.
• Code generation · same family, different data.
• Summarization · T5, BART — encoder-decoder Transformers.