Interactive Explainer

Information Theory for ML, by Hand

Drag a categorical distribution and watch entropy fall as it sharpens. Drop a model next to data and see cross-entropy split into entropy + KL. Toggle between forward and reverse KL fitting a Gaussian to a bimodal target. Build mutual information from a 2-D scatter. Every quantity recomputed live as you drag.

~20 min Probability · Information Theory · ML

Prelude

Why every loss in ML traces back to a log

Cross-entropy loss. KL divergence in VAEs. InfoNCE in contrastive learning. The "information bottleneck". Every one of these is the same handful of quantities recombined. Once you have intuition for surprise, entropy, and KL, the whole modern ML toolkit reads like one language.

Five quantities, six interactives. Each recomputes live.

The whole article in one slogan. Surprise is a log; entropy is its average; cross-entropy adds a model; KL measures the gap; mutual information measures dependence. Everything else is plumbing.

Step 1

Surprise, by the bit

The information content of an outcome with probability $p$ is $-\log p$. Rare = surprising = many bits.

A coin flip with $p=0.5$ carries 1 bit. Spotting a $p=0.001$ event carries about 10 bits — ten times as informative as a coin flip. The base of the log just sets the unit (base 2 = bits, base $e$ = nats); we'll use bits.

−log₂ p as p ranges from 0 to 1. The curve goes to ∞ at impossibility and to 0 at certainty.

Step 2

Entropy = expected surprise

Entropy is the average information you'd need per outcome drawn from $P$:

Drag the bars below to change $P$. Entropy is largest when $P$ is uniform (maximally uncertain) and falls to 0 when $P$ puts all its mass on one outcome.

Drag bar tops up or down. Numbers on the right show H(P) in bits and log₂(K) (uniform max).

Step 3

Cross-entropy = entropy + KL

Now there are two distributions: $P$ (the truth) and $Q$ (your model). Cross-entropy is the average surprise you incur using $Q$ to encode draws from $P$:

The decomposition makes the geometry obvious: cross-entropy equals entropy (an irreducible cost set by $P$ alone) plus KL divergence (the wasted cost from using a wrong model). Minimising cross-entropy in classification is exactly minimising KL, since $H(P)$ doesn't depend on your weights.

Drag the orange Q bars. Blue is fixed P. The split bar on the right decomposes H(P, Q) into H(P) (blue) plus KL(P‖Q) (orange).

Step 4

Forward vs reverse KL — the personality of the loss

KL is asymmetric. $\mathrm{KL}(P\|Q)$ and $\mathrm{KL}(Q\|P)$ are different numbers and they reward different things:

Forward $\mathrm{KL}(P\|Q)$ — mode-covering. Penalises $Q$ for putting near-zero mass anywhere $P$ has mass. The fitted $Q$ stretches to cover all modes of $P$. This is the loss you minimise in maximum likelihood.
Reverse $\mathrm{KL}(Q\|P)$ — mode-seeking. Penalises $Q$ for putting mass where $P$ has none. The fitted $Q$ collapses onto a single mode. This is what variational inference (Step 5 in the Bayesian playground) optimises.

Below: a fixed bimodal $P$. We numerically minimise either forward or reverse KL over a Gaussian $Q = \mathcal{N}(\mu, \sigma)$ with respect to its parameters. Toggle.

μ = — σ = — KL = — nats

Solid teal: target P. Dashed orange: best Gaussian Q minimising the chosen KL. Forward stretches across modes; reverse picks one and stays.

Step 5

Mutual information — how much knowing X tells you about Y

For two variables jointly distributed under $p(x, y)$, mutual information is the KL between joint and product:

Equivalently, $I(X; Y) = H(Y) - H(Y \mid X)$ — the drop in your uncertainty about $Y$ when you learn $X$. $I = 0$ iff $X \perp Y$.

correlation ρ 0.7

noise scale 0.3

Left: the 2-D scatter. Right: empirical I(X; Y) computed via histogram binning, plus the analytic Gaussian I = −½ log(1 − ρ²).

Step 6

InfoNCE — a lower bound on MI you actually optimise

MI is hard to estimate directly in high dimensions. The contrastive InfoNCE loss is a tractable lower bound:

Where $f$ is a learned similarity. The bound has the form $\mathrm{MI} \ge \log K - \mathrm{InfoNCE}$ for $K$ contrasts per positive. This is why every modern self-supervised recipe (SimCLR, CLIP, contrastive sentence encoders) is quietly an MI-maximisation.

K negatives per positive 16

signal strength 2.0

InfoNCE lower bound vs the analytic MI as K and signal strength change. The bound saturates at log K.

Step 7

Five misconceptions worth fixing

Myth

"KL is a distance."
It's not symmetric and doesn't satisfy the triangle inequality. It's a divergence. If you want a true metric, use Jensen-Shannon distance: $\sqrt{\mathrm{JSD}}$.

Myth

"Cross-entropy and KL are the same loss."
Identical only when $H(P)$ is constant in your parameters — true for hard labels (one-hot $P$ has $H = 0$), false for soft labels or label smoothing where the entropy moves.

Myth

"Forward KL is always the right loss."
MLE-style maximum likelihood minimises forward KL, which is mode-covering. If your true $P$ is multimodal and your model family $Q$ is unimodal, you'll average across modes — pick the loss that matches your goal.

Myth

"Mutual information is hard to estimate."
Above ~6 dimensions, almost every neural MI estimator is biased and high-variance. InfoNCE / MINE / CLUB give loose bounds, not faithful values. Treat MI numbers in papers as suggestive.

Myth

"Information bottleneck explains generalisation."
The original IB story was beautiful but its empirical support is contested (Saxe et al. 2018). Useful as a design principle; not a proven mechanism.

Final takeaway. Five quantities — surprise, entropy, cross-entropy, KL, mutual information — underlie every loss you'll encounter in modern ML. Once you can compute any of them from any other in your head, classification, VAEs, contrastive learning and Bayesian inference all start looking like the same recipe.