Quantization & Pruning, Side by Side

The compression menu

• Quantization. Replace 32-bit floats with fewer bits (16, 8, 4, 2, 1). 2× to 32× smaller; faster inference; accuracy varies. Two recipes: post-training quantization (PTQ) applies after training (cheap, sometimes lossy); quantization-aware training (QAT) simulates the rounding during training (recovers most of the loss).
• Pruning. Zero out weights below some magnitude threshold; remove them. Two flavours: unstructured (random sparsity, needs sparse kernels to actually speed up) and structured (drop whole channels / heads / blocks, plays well with standard kernels).
• Knowledge distillation (separate article) trains a small student to imitate a large teacher. Complementary to both above.
• Low-rank / LoRA factorisation. Replace a $W$ with $BA$ where $\mathrm{rank}(BA) \ll \mathrm{rank}(W)$. Adapts to less-than-full-rank weight matrices.
• Mixed precision. Most layers int8; a few sensitive layers FP16. The 2026 production default.

Quantization, with the math you actually need

The simplest scheme — affine (asymmetric) uniform quantisation — maps a float tensor $x \in [x_{\min}, x_{\max}]$ to a $b$-bit integer:

where $s$ is the scale and $z$ the zero-point. Dequantisation: $\hat x = s \cdot (q - z)$. Round-trip error per element is uniformly distributed in $[-s/2, s/2]$, with variance $s^2 / 12$. Halving the bit-width halves $\log_2(\text{levels})$ — so each bit saves $\sim$1× memory and adds $\sim$6 dB of quantisation noise.

Two practical choices that matter more than they sound:

• Symmetric vs asymmetric. Symmetric forces $z = 0$, so $\hat x = sq$ — saves a subtraction per multiply, at the cost of wasting half the range when $x$ is one-sided (e.g. ReLU activations). Most weight quantisers are symmetric; many activation quantisers are asymmetric.
• Per-tensor vs per-channel. Per-tensor uses one $s$ for the whole weight matrix; per-channel uses one $s_c$ per output channel. Per-channel costs nothing extra at inference (the scale folds into the bias) and is essential for matrices with wide cross-channel dynamic range. Default everywhere in 2024+.
• Outliers. A single 100× outlier weight stretches the range and crushes the resolution of all other weights. Per-channel hides this for weights; activation outliers (one large attention head in an LLM) are why SmoothQuant exists.

Live: slide bits and sparsity

A trained 2-hidden-layer MLP on a 3-class 2-D dataset. The base weights are trained to convergence; we apply quantization (PTQ) and magnitude-pruning at inference time and report the accuracy.

The thresholds you should know

• int8 with PTQ. Modern CNNs and small LLMs lose < 1% accuracy. Cheap. The default first try.
• int8 with QAT. Recovers the remaining < 0.5%; sometimes better than FP32 because quantization noise acts as regularisation.
• int4 (PTQ). 1-3% drop on most CNNs; substantial on LLMs without GPTQ / AWQ / SmoothQuant tricks. Activation outliers are the dominant problem.
• int4 (QAT or GPTQ-style). Most LLMs run at int4 in production in 2026 with ~0.5% perplexity increase.
• int2 / binary. Big quality drops. Used in specialised hardware (NPUs, ASICs) where the size win is worth the accuracy loss.
• Magnitude pruning, unstructured, 50%. Most networks tolerate ~50% sparsity with negligible accuracy loss. Above ~80% the decision boundary cracks.
• Structured pruning (whole heads / channels). Stricter limit; ~25-35% before quality drops, but real speed-up on commodity hardware.
• Mixed precision (FP16 / FP8). Modern training and inference default. NVIDIA H100 / B200 FP8; Apple Silicon FP16; Tensor cores need FP16 or BF16.

GPTQ, AWQ, SmoothQuant — the LLM quantisation family

Naively running PTQ on a 70B LLM at int4 obliterates accuracy. Three techniques restore most of it; all are post-training, all are calibration-driven (use a few hundred unlabelled prompts), and all ship in popular runtimes.

• GPTQ (Frantar et al., 2022). Quantise columns of a weight matrix one at a time, propagating the residual error into the remaining (still-float) columns via a Hessian update. Close in spirit to Optimal Brain Quantisation. Quantises a 70B model to int4 in a few hours on one GPU; near-FP16 perplexity at int4.
• AWQ (Lin et al., 2023). Scale weights and activations before quantising so the most "salient" weights (those interacting with large-magnitude activations) get more resolution. Cheaper than GPTQ; often matches or beats it for int4 LLMs.
• SmoothQuant (Xiao et al., 2022). Migrates the quantisation difficulty from activations to weights — multiply weights by $s$, divide activations by $s$, equivalent computation, much better quantisation behaviour. The fix for activation-outlier-driven failures.
• K/V cache quantisation. Beyond weights, the KV cache dominates LLM memory at long context. INT8 K/V cache is now a default in vLLM, llama.cpp; FP8 K/V is appearing in 2025+.

Lottery ticket — pruning's surprise

Frankle & Carbin (2019) showed that within a dense random-initialisation network, there exists a small subnetwork (a "winning ticket") that, when trained from the same initial weights, matches or beats the dense network's accuracy. The recipe:

1. Train dense network. Save initial weights.
2. Prune by magnitude (e.g., bottom 80%).
3. Reset surviving weights to their initial values.
4. Retrain. Often hits the same accuracy as the dense network.
5. (Iterate; "iterative magnitude pruning" finds smaller tickets.)

Practical stack for shipping a small model

1. Start big. Train the largest model your data supports; you'll compress later.
2. Distil into a smaller architecture. See the KD article; a smaller student that imitates the teacher's softmax often beats a same-size from-scratch model.
3. Apply structured pruning for real speed-up on commodity hardware. 25-35% drop without quality loss.
4. QAT at int8 as the production target. Validate on a calibrated test set (see calibration).
5. Optional: int4 with GPTQ / AWQ for big LLMs where memory is the bottleneck.
6. Profile + iterate. Almost all "the model is too slow" complaints are actually data-pipeline / I/O problems, not the model. Always profile first.
7. Deployment runtimes: ONNX Runtime, TensorRT-LLM, llama.cpp, MLC-LLM, Core ML, TFLite. Each ships its own quantization tools and has its own production gotchas.

Structured sparsity that actually speeds up hardware

Unstructured 90% sparsity rarely speeds up inference on a GPU — the dense matmul kernel still runs over the zeros. Two patterns the hardware actually likes:

• 2:4 sparsity. NVIDIA Ampere+ supports a hardware-accelerated 2:4 structured sparsity: every group of 4 consecutive weights has at most 2 non-zero. Effective 50% sparsity with ~2× speed-up at no accuracy loss after fine-tune. Now the default for production inference.
• Block / N:M sparsity. Larger blocks of zeros line up with tensor-core tile sizes. Block-sparse attention, block-sparse MLP are how 2024–2026 MoEs and sparse-attention transformers (Mixtral, Llama 4) get cheap forward passes.
• Head / channel pruning. Drop whole attention heads or feed-forward channels. Trivially speeds up but coarser; tools: HuggingFace optimum, nn_pruning.

Reading list

• Jacob et al. (2018) — Quantization and training of neural networks for efficient integer-arithmetic-only inference. The Google paper that defined modern int8 quantisation.
• Frantar, Ashkboos, Hoefler, Alistarh (2022) — GPTQ: Accurate post-training quantization for generative pre-trained transformers.
• Lin et al. (2023) — AWQ: Activation-aware Weight Quantization.
• Xiao et al. (2022) — SmoothQuant.
• Frankle & Carbin (2019) — The Lottery Ticket Hypothesis.
• Han, Mao, Dally (2016) — Deep Compression. The combined pruning + quantisation + Huffman paper.
• Mishra, Latorre, Pool, Stosic, Stosic, Venkatesh, Yu, Micikevicius (2021) — the 2:4 hardware-sparsity paper.