Where we are

Last lecture: the Transformer block. Stack it, add positional encoding, mask if autoregressive.

But stacking requires inputs. And inputs are discrete symbols (characters, subwords, words), not vectors.

Four questions:

1. Why is tokenization hard?
2. How does BPE work step-by-step?
3. What are the three pretraining paradigms — BERT, GPT, T5?
4. Why does tokenization cause so many LLM bugs?

Learning outcomes

By the end of this lecture you will be able to:

1. Explain why tokenization is the "first design decision of an LLM."
2. Run the BPE algorithm on paper for 5 merges.
3. Distinguish character / word / subword / byte-BPE and pick appropriately.
4. Identify common tokenization bugs (counting letters, arithmetic, whitespace).
5. Contrast the three pretraining paradigms · BERT · GPT · T5.
6. Name the data + compute economics of a 70B-param training run.

Three failed alternatives

A concrete failure · character LMs

Train a character-level Transformer on Wikipedia. It works — GPT-like samples of readable English.

But it's expensive:

• A 1000-word page → ~6000 characters → 6000 attention positions → 36M attention entries per layer.
• Equivalent word-level model: 1000 tokens → 1M entries. 36× cheaper.

Plus · character LMs have to learn that "t-h-e" is a word, unit by unit. That's capacity wasted on a solved problem.

Smart-dictionary analogy

That's the sweet spot we'll build with BPE on the next slides.

The sweet spot · subwords

Ideal subword tokenizer:

• Common words → single tokens (cheap, frequent)
• Rare words → composition of familiar subwords (generalizable)
• No OOV — any byte sequence is tokenizable
• Vocab size tunable (typically 30k–100k)

The winning algorithm: Byte-Pair Encoding (BPE), re-purposed from 1994 data compression.

BPE merges · visual

The BPE algorithm · 7 lines

def train_bpe(corpus, n_merges):
    # 1. Split every word into characters
    tokens = [list(word) for word in corpus.split()]
    merges = []

    for _ in range(n_merges):
        # 2. Count all adjacent pairs
        pair_counts = Counter((a, b) for word in tokens for a, b in zip(word[:-1], word[1:]))
        if not pair_counts: break

        # 3. Find the most frequent pair
        best = pair_counts.most_common(1)[0][0]
        merges.append(best)

        # 4. Apply the merge across all tokens
        tokens = [merge_in_word(w, best) for w in tokens]

    return merges, tokens

At inference, apply the same merge rules in order → tokenize any new string.

Worked BPE · merge trace

BPE · the data-compression analogy

Worked BPE · "low lower newest widest"

Step 0 · split each word into characters with </w> end marker.
l o w </w>, l o w e r </w>, n e w e s t </w>, w i d e s t </w>

Step 1 · count adjacent pairs.
(l, o): 2, (o, w): 2, (e, s): 2, (s, t): 2, (w, e): 1, (e, r): 1, …
Tie among (l,o), (o,w), (e,s), (s,t). Pick (e, s).

Merge 1 · (e, s) → "es".
l o w, l o w e r, n e w es t, w i d es t

Step 2 · recount. Now (es, t): 2 is a new pair tied with (l,o), (o,w). Pick (es, t).

Merge 2 · (es, t) → "est".
l o w, l o w e r, n e w est, w i d est

Merge 3 · (l, o) → "lo". Merge 4 · (lo, w) → "low".
After 4 merges we have learned low, est as single tokens. At inference, apply merges 1–4 in order to any new word.

Worked BPE · second example

Corpus · "hug bug rug". Initial: h u g </w>, b u g </w>, r u g </w>.

Count pairs. , , each .

Merge 1 · "ug". New: h ug </w>, b ug </w>, r ug </w>.

Recount. , others 1.

Merge 2 · "ug". Final: h ug</w>, b ug</w>, r ug</w>.

The algorithm has discovered the reusable suffix ug and the morphologically meaningful unit ug</w>.

Why byte-level BPE is the default

Two breakthroughs GPT-2 introduced:

1. Start from bytes (0–255), not Unicode characters. Every possible string becomes tokenizable, including emojis, foreign scripts, binary garbage.
2. Pretokenize by regex before BPE, to avoid crossing word boundaries ("New York" stays as two separate merge chains).

Llama, GPT-*, Mistral, Claude all use byte-level BPE with minor tweaks. SentencePiece is the same idea packaged for cross-language training.

Tokenizer comparison · same sentence, different counts

Three BPE variants you will meet

Tokenization gotchas · real LLM failures

Three families · one architecture

BERT vs GPT · side-by-side

BERT · the cloze-test analogy

This makes BERT a strong encoder · ideal for tasks where you need a representation of the whole sentence (classification, retrieval, NER). It's bad at generating text · because it never practices producing tokens one-by-one.

BERT · the cloze-test math, step by step

1. Sentence. .
2. Mask token 4. .
3. Goal. Predict the original token from context. We want to maximize .
4. Loss. Standard trick · negative log probability:

5. Total. Sum over the ~15% masked positions:

The model sees the whole sentence (no causal mask) → rich bidirectional context.

Worked numeric · BERT loss for one mask

Input · …sat [MASK] the…. Correct answer · "on".

Transformer outputs logits over vocab at the [MASK] position:

• , , , …

Softmax · , , , …
.

Loss · .

If the model were more confident (), loss would be — model rewarded for confidence on the right answer.

Great for: classification, NER, retrieval (embeddings).
Bad for: generation — can't autoregressively extend.

BERT · why mask 15%?

• Mask too few · most sequences see no loss signal → slow training.
• Mask too many · target is too hard, context too sparse.

15% was found empirically. Of those 15%:

• 80% actually replaced by [MASK]
• 10% replaced by a random token (adds noise, helps robustness)
• 10% left unchanged (so the model can't cheat by ignoring unmasked positions)

GPT · the smartphone-keyboard analogy

GPT · CLM math, step by step

Sentence · . Break into prediction problems:

1. Given <s>, predict "The". Loss · .
2. Given "The", predict "cat". Loss · .
3. Given "The cat", predict "sat". Loss · .

Total · sum over all positions:

Causal attention mask · model can only look backward.

Worked numeric · GPT loss at one step

Context · "The cat …". Correct next word · "sat".

Logits over vocab · , , , …

Softmax · , , , …
.

Loss at · .

Total sentence loss · sum these up across every position. A 2048-token window gives 2048 little training problems for free, every step.

Great for: generation, chat, code, anything where you produce text one token at a time.
Bad for: bidirectional understanding (but at scale, GPT-3+ closed this gap).

GPT · why causal loss is so rich

One tiny objective — predict the next token — forces the model to reason about:

• Syntax · closing brackets, matching tenses, agreement
• Semantics · what words follow each other meaningfully
• World knowledge · "The capital of France is …" requires a fact
• Reasoning · "If all A are B, and x is A, then x is …" requires logic
• Style · code continuations, formal-register, poetry

This scale-and-generality combo is why next-token prediction — despite looking trivial — ended up subsuming most of NLP.

T5 · encoder-decoder · text-to-text

Frame every task as text-to-text:

"translate English to German: The house is wonderful."
   → "Das Haus ist wunderbar."

"summarize: ‹paragraph›"
   → ‹summary›

"question: ‹q›  context: ‹c›"
   → ‹answer›

Survives in some translation pipelines. But for pure generation, decoder-only (GPT pattern) won.

Scaling · which paradigm won?

Why pretraining works so well

Predicting the next token on a trillion-token corpus forces the model to learn:

• Grammar — what syntactic constructions are valid
• Facts — what follows "The capital of France is"
• Reasoning patterns — what tokens usually complete "if A then"
• Style — code, poetry, legal writing, instructions

Pretraining data · where does 15T tokens come from?

Data quality beats quantity

Phi-3 (2024) · trained on 3T "textbook-quality" tokens (heavily filtered + synthetic); matched Llama-2 7B trained on 2T web tokens.

Compute economics · where the 6 comes from

The famous LLM rule of thumb: training FLOPs where = parameters, = training tokens.

Why 6? Per token:

1. Forward pass. Most compute is the FFN's two big matmuls. ≈ FLOPs per token.
2. Backward pass. Standard rule · ~2× the forward cost. ≈ FLOPs per token.
3. Total per token · .

Multiply by all tokens in the dataset:

Worked numeric · 70B model training cost

One training run of a 70B-parameter model on Chinchilla-optimal T tokens:

• FLOPs
• Time · ~25 days on 4k A100 GPUs (at ~150 TFLOPS / GPU effective utilization)
• Cost · ~$5M at commercial GPU-hour rates

Smaller check · Llama 2 7B. , :
.

Llama 3 70B · reportedly ~$80M including experiments. GPT-4 class · ~$100M+ per training run.

Scaling recap · 5 years of LLMs in one chart

Fine-tuning · from pretrained to useful

Pretrained models are knowledgeable but not steerable. To make them follow instructions, chat, or specialize on a task:

1. Supervised fine-tuning (SFT) on instruction-response pairs.
2. LoRA / QLoRA — parameter-efficient fine-tuning (next lecture).
3. RLHF / DPO — align with human preferences (L16).

The pretrained model is the brain. Fine-tuning is how you train it to do what you want.