Dropout Playground

Silence neurons at random

Dropout was Hinton's 2012 trick: on every training forward pass, randomly set a fraction of the hidden activations to zero. A unit survives with keep probability $1-p$ and is dropped with probability $p$. Write the per-unit switch as a Bernoulli variable $m_i$ and the dropped activation as $\tilde h_i$:

Two things are happening at once. The mask $m_i$ zeroes out a random subset of units. The factor $\tfrac{1}{1-p}$ rescales the survivors so the layer's expected output does not change — this is the "inverted dropout" convention PyTorch and every modern framework use, and it is exactly why the test-time path can be the plain identity. Play with both effects below.

One mask per forward pass

A tiny MLP: four inputs, ten hidden units, one output. The hidden activations $h_i$ are fixed (the weights don't change here) — only the dropout mask is resampled. In train mode each surviving unit is scaled by $\tfrac{1}{1-p}$; dropped units and their edges grey out. In eval mode dropout is off: all units fire, nothing is scaled.

Why the 1/(1−p) rescale?

Without rescaling, a layer that keeps only a fraction $1-p$ of its units at train time would fire much weaker than the full layer at test time — the distributions the next layer sees would not match. Inverted dropout fixes this by construction. Because $\mathbb{E}[m_i] = 1-p$,

so the expected activation is unchanged, and therefore the expected output is exactly the deterministic eval output:

That equality is what the orange running-mean line demonstrates above. The price you pay is variance — a single masked pass is a noisy estimate of $y^{*}$, and the noise grows sharply as $p \to 1$:

Why silencing neurons regularizes

The ensemble view

A hidden layer of $N$ units has $2^{N}$ possible on/off masks. Each minibatch trains one randomly-drawn thinned network, but all of them share the same weights. Over training you are implicitly fitting an exponential ensemble for the price of a single model. At test time you drop the mask and use the full network, which approximates the geometric-mean prediction of that whole ensemble — ensembling for free.

The co-adaptation view

Without dropout, unit $j$ can learn to lean on unit $k$ always being present — a brittle conspiracy that fits training noise. Since any teammate might vanish on the next pass, dropout forces each unit to be useful on its own and spreads the representation across many units instead of a fragile few. The result is a smoother function that generalizes better.

Where dropout lives in 2026

• Transformers — p = 0.1 after attention and the FFN. Still the default in most training recipes.
• Modern CNNs — usually 0. BatchNorm plus heavy augmentation carry the regularization budget.
• LoRA fine-tuning — typically 0 or 0.05. Adaptation data is scarce, so over-regularizing hurts.
• Classical MLPs — p = 0.5 between hidden layers. The original 2012 regime, and the one in the playground above.

Variants worth knowing

• SpatialDropout / DropBlock. Dropping individual pixels does little in CNNs because neighbours are correlated. SpatialDropout drops whole feature maps; DropBlock (Ghiasi et al., 2018) drops contiguous square regions — both far stronger for conv nets than vanilla dropout.
• DropPath / stochastic depth (Huang et al., 2016). Randomly drop entire residual blocks. Standard in very deep ResNets, EfficientNet and modern ViTs; it ensembles over network depth.
• DropConnect (Wan et al., 2013). Drop individual weights instead of activations. Higher variance per minibatch; rarely used because plain dropout is cheaper.
• Variational / MC dropout (Gal & Ghahramani, 2016). Reuse the same mask across time steps, or keep dropout on at inference to sample an approximate posterior for uncertainty.

MC dropout — dropout for uncertainty

Gal & Ghahramani (2016) showed that leaving dropout on at inference turns a deterministic net into an approximate Bayesian one: each masked pass is a sample from a posterior. Run $K$ forward passes and the spread of predictions is a cheap uncertainty estimate — exactly the blue-dot scatter above, now read as a posterior rather than as noise to average away.

It is cheap and easy, but MC-dropout uncertainties are usually under-confident and rarely beat deep ensembles. Treat it as a baseline; reach for ensembles when calibration matters, and pair either with temperature scaling or conformal prediction for confidence you can actually act on.