The Posterior, Built Up

Prelude

From "the right answer" to a distribution over answers

Maximum likelihood gives you one number for an unknown parameter $\theta$ — the value most consistent with the data. The Bayesian alternative says: the data don't pick a single $\theta$, they update your belief over all possible values. That belief is a probability distribution; the recipe for updating it is one line of math:

Three ingredients: the prior $p(\theta)$ (what you believed before), the likelihood $p(\mathcal{D}\mid\theta)$ (how the data depend on $\theta$), and the posterior $p(\theta \mid \mathcal{D})$ (your updated belief). The denominator is just a normalising constant that makes the posterior integrate to 1; we'll mostly ignore it.

The whole rest of this article is one slogan rendered in nine increasingly hard ways:

The slogan. posterior $\propto$ prior $\times$ likelihood. That's everything. All the hard machinery (MCMC, VI, GPs, Bayesian deep learning) is just how we do that one product when we can't write it down in closed form.

Step 1

Prior × likelihood = posterior, by hand

Easiest setting: a single unknown mean $\theta$ of a Gaussian with known variance $\sigma^2$. Conjugate prior is also Gaussian. The posterior is closed-form:

Drag the prior with the slider below; click on the data axis to add observations (or use the buttons). The three curves recompute live: the prior in cool blue, the likelihood as a function of $\theta$ in warm orange, and the posterior in deep teal. The posterior is the prior squeezed by the likelihood — literally the pointwise product, normalised.

prior mean 0.0

prior std 1.0

obs std σ 1.0

Click the axis to add a data point. Data points show up as ticks on the bottom rule.

n = 0 posterior μ̂ = —, σ̂ = —

Read the curves. With no data the posterior is the prior. Add one sample; the posterior starts shifting toward it. As you keep adding, the posterior tightens around the data mean and the prior loses influence. That's it — the entire Bayesian ritual.

Step 2

How fast does data sharpen the posterior?

The posterior precision (inverse variance) is the sum of the prior precision and the data precision:

Two clean consequences. First, posterior variance falls like $1/n$ — classical "data sharpens beliefs" but with a principled rate. Second, the prior contributes a fixed amount of pseudo-data: a tight prior with $\sigma_0 = 0.1$ is "worth" $\sigma^2 / \sigma_0^2 = 100$ data points for $\sigma=1$.

Posterior std vs sample size for three prior strengths. Theory line is $\sigma_n = (\tau_0 + n/\sigma^2)^{-1/2}$.

The horizontal asymptote at $1/\sigma_0$ goes away as $n$ grows; eventually the data precision dominates whatever prior you brought.

Step 3

The conjugate coin (Beta-Binomial)

Bernoulli observations, parameter $\theta = $ probability of heads, prior $\theta \sim \mathrm{Beta}(\alpha, \beta)$. After $k$ heads in $n$ flips:

Conjugacy means "the posterior stays in the same family" — here Beta. Update is just adding counts to the parameters. Click the buttons below to flip a real coin and watch the Beta posterior reshape live.

prior α 1.0

prior β 1.0

Beta(α, β) prior in blue, Beta(α+k, β+n−k) posterior in teal. Coin's true bias = 0.65 (drawn for the random buttons).

k / n = 0/0 posterior mean = —, 95% CI = —

Why Beta(1, 1) is uniform. Setting $\alpha = \beta = 1$ gives the flat prior on [0,1]: every coin bias is a priori equally likely. With this prior the posterior mean after $k$ heads in $n$ flips is $(k+1)/(n+2)$ — Laplace's rule of succession. That is what your stats teacher mumbled about why we add 1 to the numerator and 2 to the denominator in small-sample probabilities.

Step 4

Posterior predictive ≠ plug-in MAP

The posterior is the right object for the parameter. But downstream we usually care about predicting new data. There are two ways to do that:

Plug-in MAP / MLE. Pick one $\theta$ (the maximum) and predict $p(x_{\text{new}}\mid\hat\theta)$.
Posterior predictive. Average over every $\theta$ weighted by the posterior: $\;p(x_{\text{new}}\mid\mathcal{D}) = \int p(x_{\text{new}}\mid\theta)\,p(\theta\mid\mathcal{D})\,d\theta$.

Plug-in is faster but lies about uncertainty. The posterior predictive is wider whenever you don't yet know $\theta$ precisely — precisely the regime where calibration matters most. Watch the gap shrink as $n$ grows.

For the same Gaussian-Gaussian setup as Step 1 with a fresh sample drawn from the predictive distributions: dashed = plug-in MAP, solid = full posterior predictive.

n samples 3

obs std σ 1.0

What this costs you in practice. A neural net that outputs softmax over classes is a plug-in: it pretends the weights are fixed. When the weights are still uncertain (i.e., always, on real data), the predictive distribution should be wider. That gap shows up as an overconfident, miscalibrated model — the bug Bayesian deep learning is trying to fix.

Step 5

What if the posterior isn't a nice family?

Conjugate priors give us closed-form posteriors. Real posteriors — from a neural net, a hierarchical model, a non-Gaussian likelihood — usually aren't conjugate. Three families of algorithms cover all of practice:

Grid evaluation. Compute the unnormalised posterior on a grid; normalise. Exact, dies above ~3 dimensions.
MCMC (we use Metropolis-Hastings). Random walk on $\theta$ that converges in distribution to the posterior. Exact in the limit, finite-sample-noisy.
Variational inference (VI). Pick a tractable family (we use a Gaussian) and fit it by minimising the KL divergence to the true posterior. Fast but biased — especially for multimodal targets.

Below: a deliberately cooked, bimodal, non-conjugate posterior. Grid runs once. The MCMC chain walks live; you can watch samples slowly fill out both modes (or get stuck in one). The VI fit is a single Gaussian dropped on top — almost always centered on one mode and miscalling the other.

Solid = true (grid) posterior. Histogram bars = MCMC samples. Dashed orange = best-fit Gaussian (VI).

step size 0.30 samples = 0 accept rate = —

Reading the picture. Grid is exact but exhaustive. MCMC histogram converges to grid but takes many samples (and the right step size: too small → no movement, too large → all rejections). VI is fast and pleasing-looking but quietly wrong wherever your true posterior isn't unimodal-Gaussian. This is the single most consequential uncertainty in modern Bayesian deep learning.

Step 6

Bayesian linear regression — uncertainty that bows out

One last conjugate model with a famous picture. Linear regression with Gaussian prior on the weights and Gaussian observation noise has a Gaussian posterior over weights, and a Gaussian posterior predictive at every input $x$:

Click anywhere to add a data point. The fit line and its uncertainty band recompute live. Notice the band: thin where you have data, fat where you don't. That is what honest uncertainty looks like, and it is exactly what Gaussian processes generalise.

Click to add a data point; right-click (or shift-click) to remove. Solid blue: posterior mean. Shaded teal band: ±2 posterior-predictive σ.

prior weight std 1.0

obs noise σ 0.4

n = 0

The big takeaway. The uncertainty band tells you where the model is allowed to be wrong. A model that hands you a point estimate where you have no data is lying about what it knows. Bayesian inference gives you the right answer to "how unsure am I?" for free, as a side effect of doing the math correctly.

Step 6 ½

Hierarchical models — partial pooling, the Bayesian superpower

Suppose you have several related groups: many sites, sensors, schools, patients, batches. Each has a small pile of observations. You can fit one model per group ("no pooling") or one model for everything ("complete pooling"). Both are wrong in known ways. The Bayesian fix is partial pooling: a hierarchical prior that lets each group have its own mean while pulling outliers toward the global mean.

$\mu$ is the global mean; $\tau$ is how strongly groups are pulled toward it; $\theta_g$ is the group-specific mean. The posterior of $\theta_g$ is shrunk toward $\mu$ by an amount that depends on $\tau$ and the group's data count $n_g$.

shrinkage τ 1.00 obs noise σ 1.00 include data-poor group (n=2)

Eight groups along the x-axis. Black dots: observations. Big symbols: per-group MLE (orange ×), partial-pool posterior mean (teal ●), and complete-pool global mean (blue dashed line). Watch the data-poor group's estimate pull toward the global mean as τ shrinks.

The shrinkage formula. For Gaussian data, the posterior mean of group $g$ is a weighted average: $\hat\theta_g = w_g \bar y_g + (1 - w_g)\,\mu,$ $\;w_g = \frac{n_g/\sigma^2}{n_g/\sigma^2 + 1/\tau^2}.$ Big $n_g$ or small $\tau$ ⇒ trust the group; small $n_g$ or big $\tau$ ⇒ trust the global mean. Partial pooling interpolates the two extremes per group.

Why this matters: data-poor groups borrow strength from data-rich ones without being forced to agree with them. James & Stein, 1961, proved that this estimator dominates the per-group MLE in mean squared error — in three or more dimensions, no-pool is provably suboptimal.

Step 7

Five things people get wrong

Myth

"Bayesian = subjective, frequentist = objective."
Both pick assumptions: frequentists pick a sampling model and a procedure (significance level, estimator); Bayesians pick a sampling model and a prior. Bayes makes the assumption explicit, which is the actual difference.

Myth

"With enough data the prior doesn't matter."
True for low-dimensional, well-identified parameters (Step 2 made this concrete). False in high-dimensional models, hierarchical models, or any setting where the likelihood is flat in some directions — which is most of modern deep learning.

Myth

"MCMC always converges to the truth."
In the infinite-sample limit, yes. With finite chains and poor proposals (Step 5), MCMC spends a long time in a single mode, returns a confidently wrong histogram, and tells you it converged. Always run multiple chains.

Myth

"VI is just a faster MCMC."
VI is fundamentally biased: it returns the closest member of your variational family, even if no member of the family looks like the truth. On multimodal posteriors a Gaussian VI fit can put zero mass on the second mode while still claiming to have "converged".

Myth

"A Bayesian neural network is just one trained with a prior."
The prior is the easy part. The hard part is getting a calibrated posterior over a 100M-dimensional parameter space, which neither MCMC nor VI does honestly today. SGLD, Laplace, deep ensembles, and SWAG are practical compromises — none of them is "the Bayesian answer".

Myth

"Bayesian inference is overkill if you only want a point estimate."
The posterior mean minimises mean-squared error for any loss; the MAP minimises 0–1 loss. They're often different, and pretending the MAP is the right summary has cost more papers than overfitting has.

Step 8

Where this lives in your research

Three places the Bayesian frame buys you something concrete rather than aesthetic:

Calibrated uncertainty in deployed models. A sensor model that says "I don't know" off-distribution is safer than one that says "0.97" everywhere. Reliability diagrams + temperature scaling + deep ensembles get you most of the way; full Bayesian deep learning is the principled limit.
Hierarchical models for sparse, structured data. Patients, sites, sensors, languages, individuals — anywhere you have many groups with few observations each, partial pooling beats both "one model per group" and "one model for all" by a lot. This is the Bayesian superpower for data with regional structure.
Inverse problems and physics-informed ML. Energy disaggregation, source apportionment for air quality, satellite pixel unmixing, sleep-stage estimation — the unknown is a high-dimensional latent and the forward model is partially known. Bayes is the language of inverse problems; everything else is an approximation to it.

Final takeaway. The posterior is a probability distribution. You don't need new math to reason about it — just patience to compute the product on the right side of Bayes' rule. The eight interactives above are eight versions of that one product. Everything else (MCMC, VI, GPs, BNNs) is plumbing.