Interactive Explainer

BatchNorm vs LayerNorm vs GroupNorm vs InstanceNorm

Four normalization layers, one equation shape: subtract a mean, divide by a standard deviation. The only thing that changes is which slice of the activation tensor the mean and variance are pooled over. That single design choice is why BatchNorm ruled CNNs, LayerNorm rules Transformers, and GroupNorm rescued small-batch training. Pick a norm below and watch exactly which elements share one statistic.

~12 min Normalization · CNNs · Transformers

Prelude

One equation, four slices

Take a batch of activations shaped $(N, C, H, W)$ — $N$ examples, $C$ channels, and an $H\times W$ spatial map per channel. Every normalization layer applies the same transform to each element, then a learned scale $\gamma$ and shift $\beta$:

\hat{x} = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta .

The mean $\mu$ and variance $\sigma^2$ are what differ. Each method sums over a different set of axes — a different slice of the tensor — and therefore produces a different number of independent statistics:

BatchNorm · pool over $N, H, W$ — one $(\mu,\sigma^2)$ per channel.
LayerNorm · pool over $C, H, W$ — one per sample.
InstanceNorm · pool over $H, W$ — one per (sample, channel).
GroupNorm · pool over a channel group $+\,H, W$ — one per (sample, group).

The Lab

Which elements share one statistic?

Below is one activation tensor. Rows are the $N$ examples in the batch, columns are the $C$ channels, and each little tile is that channel's $H\times W$ spatial map (here $2\times2$). Choose a normalization scheme — the highlighted cells are exactly the elements averaged together for one $(\mu,\sigma^2)$. Click any tile to move the reference cell and see which slice it belongs to.

Batch size $N$ 4

Channels $C$ 6

Channels / group $G$ 3

Each tile is a channel's 2×2 spatial map. Blue tiles are pooled into one mean & variance.

averaged together (one $\mu,\sigma^2$) other elements reference cell · click to move

Mean $\mu$ of highlighted —

Variance $\sigma^2$ of highlighted —

Elements pooled — —

Try this. Keep the reference tile fixed and flip between BatchNorm and LayerNorm. Watch the mean pill: BatchNorm pools a whole channel across the batch, so its statistic barely moves when you click a different sample — but LayerNorm re-computes a fresh mean for each sample, so it jumps. That batch-vs-sample split is the entire story: BatchNorm couples examples together (great with big batches, fragile with tiny ones); LayerNorm treats every example alone (perfect for variable-length Transformer sequences). Now drag $G$ under GroupNorm from $1$ up to $C$ — at $G=1$ it is InstanceNorm, and at $G=C$ it is LayerNorm.

The Payoff

Why each wins where it wins

CNNs with large batches · BatchNorm. A channel is a repeated feature detector applied at every spatial location and every image, so $N\!\cdot\!H\!\cdot\!W$ samples give a stable estimate of its mean. The catch: the statistic depends on the whole batch, so a batch of 2–4 (common in detection/segmentation) makes it noisy, and train vs eval must swap in running averages.
Transformers & RNNs · LayerNorm. Sequence lengths vary and batches can be size 1, so no batch dependence is essential. LayerNorm normalizes each token's feature vector on its own — identical behaviour at train and eval, no running stats.
Small-batch vision · GroupNorm. It keeps LayerNorm's batch-independence but splits channels into groups, recovering some of the per-channel structure BatchNorm exploited. It matches BatchNorm's accuracy on ImageNet at batch size 2, where BatchNorm collapses.
Style transfer · InstanceNorm. Normalizing each image and channel independently strips out per-image contrast and colour statistics — exactly the "style" you then want to repaint, which is why it is the default in style-transfer and many GAN generators.

And RMSNorm? The modern Transformer favourite is just LayerNorm with the mean-centering dropped: $\hat{x} = \gamma \cdot x / \sqrt{\overline{x^2} + \epsilon}$. Same slice as LayerNorm (across $C$), but it skips subtracting $\mu$ — one fewer statistic and one fewer parameter for near-identical accuracy. Every method here is a variation on the same question you were just clicking through: over which axes do we pool?