Interactive Explainer

The KV-Cache, Made Concrete

Every token an LLM generates re-reads every key and value from all previous tokens. Caching them is mandatory — and the cache quickly dwarfs the weights. Slide the model, the context, and the batch below and watch the moment the KV-cache overtakes the parameters. This is why “LLM inference is memory-bound, not compute-bound.”

~11 min Inference · LLM systems

Prelude

Why the cache has to exist

To predict token $t{+}1$, self-attention compares the current query against the key of every earlier token and mixes in their values:

\mathrm{attn}(q_t) = \sum_{i \le t} \mathrm{softmax}\!\big(q_t \cdot k_i / \sqrt{d_h}\big)\, v_i .

The keys $k_i$ and values $v_i$ of past tokens never change. Recomputing them at each step would make generation cost $\mathcal{O}(n^2)$ in time for no reason. So we cache them: after a token is processed, its $k$ and $v$ (in every layer, every head) are stored and re-used forever. That trade — spend memory to save compute — is what makes autoregressive decoding fast, and what makes it memory-bound. Every new token reads the entire cache back out of HBM; the arithmetic is trivial next to the bytes moved.

Step 1

The one formula that governs everything

For a single token, in a single layer, one attention head stores one key vector and one value vector, each of dimension $d_h$. Multiply out over the whole model and you get the size of the cache per token:

\underbrace{\text{bytes}_{\text{token}}}_{\text{per token}} = \underbrace{2}_{K \text{ and } V}\cdot\; L \;\cdot\; n_{kv} \;\cdot\; d_h \;\cdot\; p

$L$ — number of transformer layers (the cache is stored once per layer).
$n_{kv}$ — number of key/value heads. Under multi-head attention this equals the query heads; grouped-query and multi-query attention shrink it.
$d_h$ — dimension of each head.
$p$ — bytes per number (FP16 = 2, INT8 = 1).
The factor $2$ is because we store both a key and a value.

Multiply by the context length and the batch size (concurrent sequences) and you have the total footprint: $\;\text{bytes}_{\text{total}} = \text{bytes}_{\text{token}} \cdot \text{context} \cdot \text{batch}.$ Nothing here depends on the size of the MLP or the vocabulary — the cache is a pure function of the attention shape. Time to make it move.

The Lab

The memory-cliff calculator

Set a model with the sliders, or load a real one with a preset. The chart plots two things against context length: the flat model weights and the rising KV-cache. Where the cache line overtakes the weights is the memory cliff — past it, your GPU is spending more on remembering the conversation than on the model itself.

Model shape

Layers $L$ 80

Query heads $n_h$ 64

Head dim $d_h$ 128

Attention type → sets KV heads $n_{kv}$

GQA key/value heads 8

KV-cache dtype → bytes per number $p$

Effective KV heads: 8 — each shared by 8 query heads.

Context length 8,192 tok

Batch size (concurrent users) 8

Per token — all layers & KV heads

Total KV-cache — context × batch

Model weights — — params, FP16

KV ÷ weights — —

Flat blue band: model weights (fixed). Orange area: KV-cache, growing linearly with context at the current batch. The dashed vertical line is the cliff where they are equal; the solid orange line marks your current context. Horizontal dashes are 80 GB GPU increments.

Total footprint (weights + current KV-cache) laid against 80 GB GPU boundaries.

Try this. Load Llama-3 70B, then drag batch size up from 1. At batch 1 the cache is a footnote next to the 130 GB of weights. But KV scales with every concurrent user, so by batch 32 the orange area has climbed past the blue band — the same model now needs multiple GPUs just to hold conversations in flight. Now switch the same model to MHA: with $n_{kv}$ jumping from 8 to 64, the cliff arrives $8\times$ sooner. That single design choice is why every modern model uses GQA.

Step 2

PagedAttention: stop reserving what you don't use

There is a second, subtler waste. A naive server does not know how long a reply will be, so it reserves a contiguous buffer for the maximum context of every sequence up front. Most replies are short, so most of that buffer sits empty — internal fragmentation. vLLM's PagedAttention borrows the operating-system trick of paging: memory is a pool of small fixed blocks, and a sequence grabs blocks only as it grows. Toggle between the two and watch the wasted (hatched) memory collapse.

Allocation strategy

Concurrent sequences 6

Block size 128 tok

Each cell is one block. Solid = tokens actually stored; hatched = reserved but empty.

stored tokens reserved & wasted

Memory used — blocks holding real tokens

Reserved total — blocks paid for

Utilisation — used ÷ reserved

Why it matters. Naive reservation routinely wastes 60–80% of KV memory. By paging, vLLM keeps utilisation near 100%, which means far more sequences fit on the same card — higher throughput from the exact same hardware. Smaller blocks waste less at the tail but add bookkeeping; 16–128 tokens per block is the usual sweet spot.

Step 3

The three escape hatches

Every term in $2 \cdot L \cdot n_{kv} \cdot d_h \cdot p$ is a lever, and the field has pulled each of them:

shrink n_kv

Grouped-query attention

Let several query heads share one key/value head. GQA(8) cuts $n_{kv}$ from 64 to 8 — an $8\times$ smaller cache with negligible quality loss. MQA takes it to the extreme with a single KV head.

shrink p

INT8 / FP8 KV-cache

Store keys and values in 8 bits instead of 16. Halves the bytes, often for free; occasionally a small accuracy nudge. Toggle it in the Lab and the cliff moves twice as far right.

stop wasting

PagedAttention

Allocate KV in fixed blocks instead of one giant contiguous buffer per sequence. No fragmentation, no over-reservation — the trick at the heart of vLLM's throughput.

Where each model lands

Model	Attention	KV heads	KV / token (FP16)
Llama-2 70B	MHA	64	`~2.5 MB`
Llama-3 70B	GQA	8	`~320 KB`
Mistral 7B	GQA	8	`~128 KB`
Multi-query (MQA)	MQA	1	`~40 KB`

Takeaway. The KV-cache is the hidden tax on long context and high concurrency. Its size is set entirely by the attention shape — $2 \cdot L \cdot n_{kv} \cdot d_h \cdot p$ — and it grows with both the length of each conversation and the number of them at once. Serving LLMs at scale is, to a first approximation, the art of keeping this number small.