BPE Tokenizer Step by Step

Tokenization is invisible until it breaks. LLMs don't see characters — they see whatever subwords BPE learned to carve out. This page grows a tokenizer merge by merge, so you can watch it glue common pairs and end up with tokens like "ization", "tion", or " the".

Byte-Pair Encoding was invented in 1994 for data compression. It showed up in NLP in 2015 (Sennrich et al.), and became the standard tokenizer for GPT-2 (2019) and everything after. The algorithm is tiny: repeatedly merge the most frequent adjacent pair, stop at target vocab size.

The playground

Try this: Press "Initialize (chars)", then "Merge 10". Watch the vocabulary grow from letters into meaningful subwords. Notice how the tokenizer discovers suffixes ("ing", "ed") and common function words ("the") without any linguistic input — just counting.

Why this matters. Every LLM's vocabulary is frozen at training time — GPT-2 has 50,257 tokens; Llama has 32k. When you type a rare word, it splits into whatever subwords were learned. This is why LLMs spell "strawberry" as "straw" + "berry" and sometimes miscount its letters.

Tokenization across models

GPT-2, GPT-3 — byte-level BPE, 50,257 tokens.
BERT — WordPiece (a variant of BPE), 30,522 tokens.
Llama, Mistral — SentencePiece + BPE, 32k tokens.
Claude, GPT-4 / GPT-4o — proprietary, ~100k–200k tokens.
Gemini, Qwen — proprietary; large vocabularies tuned for multilingual + code.

The three tokeniser families

BPE (Sennrich et al., 2015). Start with characters; repeatedly merge the most frequent pair. Outputs a single deterministic token sequence for any string. Used in GPT-2/3/4, Llama, Mistral.
WordPiece (Schuster & Nakajima, 2012; in BERT, 2018). Similar to BPE but the merge criterion is mutual information / likelihood gain, not raw frequency. Tokens that increase the data likelihood the most are merged. Marginally better at handling rare words.
SentencePiece (Unigram LM; Kudo, 2018). Treats tokenisation as a unigram language model. Starts from a huge vocabulary, iteratively removes tokens that hurt the corpus log-likelihood the least, until the target vocab size is reached. Produces probabilistic tokenisation (each text has multiple valid token sequences with different probabilities — useful for subword regularisation).

Byte-level BPE — handling any Unicode

A pure-BPE tokeniser breaks on text containing characters it never saw at training. GPT-2 solves this with byte-level BPE: operate on bytes (256 possible) instead of characters, so any Unicode string decomposes into a byte sequence the tokeniser knows. Adds a small overhead (Cyrillic / CJK get ~2× more tokens than Latin) in exchange for never crashing on rare scripts or emoji.

The 'strawberry has 3 r's' phenomenon

LLMs often miscount letters in their own output. The reason is tokenisation: "strawberry" is typically split into "straw" + "berry" (or just one token!). The model never sees individual r's; it sees subword tokens, none of which is "r". To answer "how many r's in strawberry?" the model has to decompose its own tokens — which it does imperfectly because the decomposition never appeared during training.

The same explains: weird arithmetic on numbers tokenised as one piece, off-by-one errors on reversing text, and inability to spell out names character by character. The fix in 2024+ is finer-grained number tokenisation (single-digit tokens for numbers) plus character-level data mixed in during training.

Reading list

Sennrich, Haddow, Birch (2015) — Neural Machine Translation of Rare Words with Subword Units. The paper that brought BPE to NLP.
Kudo (2018) — Subword Regularization. The Unigram LM tokeniser.
Kudo & Richardson (2018) — SentencePiece. The implementation that ships with Llama, T5, etc.
Wu et al. (2016) — Google's Neural Machine Translation System. WordPiece in production.
Tokenizer playground — tiktoken (OpenAI), tokenizers (HuggingFace).