Learning outcomes

By the end of this lecture you will be able to:

1. Explain why LLM inference is memory-bound (not compute-bound).
2. Compute the KV-cache size for Llama 70B at 32k context.
3. Describe paged attention and why vLLM needed it.
4. Pick a quantization level (FP16 / INT8 / INT4 / INT2) for a deployment.
5. Explain FlashAttention tiling in 2 sentences.
6. Describe speculative decoding and its accept-rate dependency.

Where we are

Training a 70B LLM costs ~$100M. But that's done once. Inference runs every request, every user, every day — and it's where models earn their keep.

Four questions:

1. Why is LLM inference memory-bound, not compute-bound?
2. What is the KV-cache and how do we manage it?
3. What is quantization and how low can we go?
4. What are FlashAttention and speculative decoding?

Reading vs writing · the inference analogy

That's why decode is much slower per token than prefill · we can't parallelize across the future. Almost every inference optimization (KV cache, speculative decode, FlashAttention) targets the decode phase.

Prefill vs decode · the kitchen analogy

Decode · why it's memory-bound

For one token of decode on a 70B model the GPU must:

1. Fetch all 70B weights from HBM. At 2 bytes (BF16) = 140 GB.
2. Fetch the KV-cache for the context (>10 GB at 32k context).
3. Do the math · matrix–vector products. Tiny compared to the data movement.

NVIDIA A100 · ~1.5 TB/s HBM bandwidth → moving ~150 GB takes ~0.1 s. The math finishes in a fraction of that. The GPU spends most of its time waiting for data.

Prefill is different · the whole prompt is processed together as a big matrix–matrix product → the GPU's FLOPs are saturated. Compute-bound.

Modern inference servers (vLLM, TGI) optimize the two phases separately.

The KV-cache explained

Cost with vs without cache

Why caching KV works

Observation · attention at step computes · we need every past K, but doesn't change once token is generated.

Memory grows linearly with context, but saves an order of magnitude in compute. The reason KV-cache is the first thing any LLM inference stack implements.

KV-cache math · Llama 70B

With MHA (64 heads, no GQA) that would be 84 GB — larger than the weights themselves.

Paged attention · the parking-lot analogy

For LLMs · "vehicles" = ongoing requests, each needing variable-length KV cache. Pages let us serve many short prompts in the memory that previously held one long-prompt KV cache. ~4× throughput in vLLM.

Paged attention · vLLM's big idea

Kwon et al. 2023 · vLLM paper.

Problem · KV-cache memory is allocated contiguously; long-context requests waste space.

Solution · paged attention — split the KV-cache into fixed-size pages, managed like virtual memory. Each request uses only as many pages as it needs.

The quantization ladder

INT8 · the map-scale analogy

INT8 · derive the formula step by step

Goal · convert FP32 weights to 8-bit integers in .

Step 1. Find max absolute value in the channel: .
Step 2. Map the largest weight to the largest integer (127):

Step 3. Quantize each weight:

Step 4. Reconstruct at inference: .

Store the int8 array + a single FP32 scale per channel.

INT8 · worked numeric

Convert from FP32 to INT8.

1. Max abs. .
2. Scale. .
3. Quantize.

Stored · 4 INT8 (4 bytes) + scale FP32 (4 bytes) = 8 bytes vs 16 bytes (4× FP32). 50% saving.

Reconstruction. (vs original 0.5). Max error ~0.001 — quality drop barely measurable.

PyTorch · torch.quantization.quantize_dynamic or bitsandbytes.

Quantization · arithmetic in pictures

Worked example · quantize a single weight row

Reconstruction at inference · → . Max error . Roundoff is small; quality drop on benchmarks barely measurable.

INT4 and below · GPTQ / AWQ

At 4 bits per weight, naive quantization breaks. Two successful tricks:

• GPTQ (Frantar 2022) · post-training per-layer quantization that minimizes reconstruction loss.
• AWQ (Lin 2023) · protects the ~1% of weights that activate on important inputs.

Both work well at 4-bit; AWQ edges out GPTQ at extreme compression (3-bit, 2-bit).

The attention memory problem

Naive attention materializes the attention matrix:

scores = Q @ K.T                 # [B, H, N, N]   ← huge for long context
weights = scores.softmax(dim=-1)
out = weights @ V                 # [B, H, N, d_h]

For , a single layer needs ~8 GB just for the softmax matrix. GPU HBM bandwidth becomes the bottleneck, not FLOPs.

FlashAttention · tiles in SRAM

FlashAttention · the giant-mural analogy

For , the attention matrix has ~1B elements → 4.3 GB per head per layer in FP32. Too big to read/write to main memory efficiently.

FlashAttention · how it works

Three ideas (Dao et al. 2022):

1. Tile · break into blocks (e.g. 256×256). Each block fits in SRAM (~20× faster than HBM).
2. Fuse · do the entire (matmul + softmax + matmul) for a block in one on-chip kernel — never write the intermediate to HBM.
3. Online softmax · math trick to update softmax incrementally as new blocks come in. Result is mathematically identical to naive — not an approximation.

Memory savings · 8192-context, FP16.

• Naive · bytes ≈ 134 MB per head/layer (must hit HBM).
• Flash · 256×256 tile · in SRAM. The 134 MB is never created.

PyTorch 2.0+ ships F.scaled_dot_product_attention — just use it.

FlashAttention · adoption

• PyTorch 2.0+ · built-in as F.scaled_dot_product_attention.
• All major inference servers use it.
• FlashAttention 3 (2024) · further optimizations for H100 Hopper.

Speculative decoding · picture

The decode speedup trick

Autoregressive generation is one token per forward pass. For a 70B model this caps your throughput.

Speculative decoding (Leviathan et al. 2022):

1. A small draft model (~1B params) quickly guesses the next tokens.
2. The big verifier model (70B) does ONE forward pass on all in parallel.
3. Accept the longest prefix where draft and verifier agree; rewind if they diverge.

Why speculative works

When the draft is right · tokens per forward pass instead of 1. 2–4× speedup.

When the draft is wrong · same cost as normal decoding (pay for one extra forward on the draft).

Net effect · draft is right ~70% of the time on typical text → ~2× speedup for free.

Distillation · the master-chef analogy

Distillation · the loss, term by term

Two losses:

Part 1 · standard CE against true labels:

Part 2 · imitate the teacher's distribution. Use KL divergence between softened distributions (temperature ):

Combine with weight :

The corrects for the gradient shrinkage that softening causes (so the two losses are comparable in scale).

Distillation · worked numeric

Image of a cat. Classes [Cat, Dog, Car].

• Teacher logits .
• Student logits .

. softmax() = . Almost no dark knowledge — student barely learns relative class structure.

. Soften logits: , .

• Soft teacher · softmax() ≈
• Soft student · softmax() ≈

Now the student learns: increase Cat, decrease Dog and Car, but keep Dog about 2× Car. Rich nuanced signal that pure hard-labels would miss.

Distillation in 2026

• DistilBERT (2019) · 40% smaller, 60% faster, 97% of BERT's quality. Still used.
• DistilLLaMA, DistilGPT-2 · similar story for generative.
• Modern practice · distill a 70B teacher into a 7B student with task-specific data · near-teacher quality at 10× cheaper inference.

What a production LLM server does

1. Batched inference · pack many user requests into each forward pass.
2. Paged attention for KV-cache memory efficiency.
3. INT8/INT4 quantization for the weights.
4. FlashAttention for attention compute.
5. Speculative decoding for throughput.
6. Continuous batching · new requests can join mid-batch without restart.
7. Streaming output · send tokens as they are generated.

Put together · ~10–50× faster and cheaper than naive implementations.

The inference frameworks

Choose by hardware + quality needs. For teaching, vLLM or llama.cpp are easiest to install.

End-to-end · optimization stack compounded

Cost economics · 2026