Knowledge Distillation, Live

Why a small model can learn from a big model better than from the data

You have a big, accurate teacher. You want a small, fast student that runs on a phone, a sensor node, an edge GPU. The naive recipe: train the student from scratch on the same data. The better recipe: train the student to imitate the teacher's softened outputs. The student often beats the data-only baseline by a lot — with the same parameters, on the same dataset.

The trick is information that's invisible in hard labels. A cat picture has a hard label "cat", but a teacher's softmax says "cat 0.93, lynx 0.04, tiger 0.02, dog 0.01". That ordering carries dark knowledge — the teacher's opinion about how classes relate. The student picks it up for free.

Soften with temperature

Standard softmax converts logits $z_c$ into probabilities $p_c$. Knowledge distillation introduces a temperature $T$:

$T=1$ is the usual softmax. Larger $T$ flattens the distribution, exposing more of the relative ordering. As $T \to \infty$ everything becomes uniform; as $T \to 0$ it becomes one-hot. The whole game is to pick a $T$ where the non-winning classes still carry useful structure.

The two losses, side by side

The student gets two supervision signals:

• Hard cross-entropy against the dataset label $y$.
• Soft KL from the teacher's softened probabilities $p^{(T)}$ to the student's softened probabilities $q^{(T)}$, both at temperature $T$.

The $T^2$ multiplier in the KD term keeps gradients on the same scale at any $T$ (Hinton et al., 2015). $\alpha$ trades off how much the student listens to data vs. teacher.

Watch the student learn

The teacher below is a fixed Gaussian-mixture classifier on a 2-D plane: smooth, accurate, but 'big' in the sense that we assume we don't get to ship it. The student is a single linear layer (just $2 \times K$ weights) — barely more than logistic regression. Train it three ways and compare.

Where does the gain come from?

Three angles on the same phenomenon:

• Label smoothing on steroids. Hard labels are one-hot. The teacher's softened probabilities are a per-input smoothing that depends on the actual geometry of the classes. Generic label smoothing replaces one-hot with a fixed mixture; KD replaces it with a learned mixture that knows your data.
• Pseudo-data outside the training set. The teacher answers the student's questions even at points the student never saw labelled — effectively giving the student a free, infinitely-large dataset (whose ground truth happens to be the teacher).
• Gradient richness. A one-hot target gives a gradient that pushes one logit up and (everything-else) down equally. A softened target gives a gradient that pushes each non-winner by a different amount — far more information per backward pass.

The variants you'll meet in papers

• Logit / response distillation. The classical recipe above. Cheapest. Works surprisingly well.
• Feature distillation. Match intermediate activations or attention maps (FitNets, Romero et al.; attention transfer, Zagoruyko & Komodakis). Useful when the student's capacity bottleneck is hidden, not in the output head.
• Self-distillation. Student and teacher share an architecture; student is initialised from a previous training run. Adds 1–3% accuracy "for free".
• Online / mutual KD. No frozen teacher; two students teach each other (Zhang et al., 2018). Useful when you don't have a pretrained big model to start with.
• Born-again networks. Distil to the same architecture; iterate. Each "generation" tends to gain a little (Furlanello et al., 2018).

Five things to remember