Interactive Explainer

MAP = Likelihood × Prior

“Add a regularizer to the loss” sounds like a hack. It isn't. Put a prior on your parameters, multiply it by the likelihood, and maximize the product — that's the MAP estimate, and the regularizer falls out automatically. This page draws all three distributions in parameter space so you can see the MAP sit between the data's answer (MLE) and the prior's pull toward zero.

~11 min Deep Learning · Regularization · Bayesian Inference

Prelude

Two peaks, one product

Take a tiny model with just two weights $\theta = (\theta_1, \theta_2)$. The likelihood $p(\text{data}\mid\theta)$ is a bump centered on the value that best fits the data — the MLE. The prior $p(\theta)$ is a bump centered at the origin, encoding “keep the weights small.” Bayes multiplies them into the posterior:

\underbrace{p(\theta\mid\text{data})}_{\text{posterior}} \;\propto\; \underbrace{p(\text{data}\mid\theta)}_{\text{likelihood}}\; \underbrace{p(\theta)}_{\text{prior}}.

Take $-\log$ of both sides and the product becomes a sum — a data-fit term plus a penalty term:

\hat\theta_{\text{MAP}} = \arg\min_\theta\; \underbrace{-\log p(\text{data}\mid\theta)}_{\text{loss}} \;+\; \underbrace{-\log p(\theta)}_{\text{regularizer}}.

A Gaussian prior gives $-\log p(\theta) \propto \lVert\theta\rVert_2^2$ — that's ℓ₂ regularization (ridge). A Laplace prior gives $-\log p(\theta) \propto \lVert\theta\rVert_1$ — that's ℓ₁ regularization (lasso). The regularizer was a prior all along.

The Lab

Parameter space, live

The square is parameter space. Blue contours are the likelihood (data fit), teal contours are the prior (centered at the origin), and orange contours are their product, the posterior. The three marked points are the MLE (data's best fit), the MAP (posterior peak), and the origin. Move the controls and watch the MAP get tugged between the data and zero.

Amount of data few

Noise level medium

Prior width $\tau$ 1.00

Prior correlation $\rho$ 0.00

Each contour joins parameter values with the same negative log density; inner contours are more plausible. Narrow $\tau$ = strong prior; more data / less noise = tighter likelihood.

Likelihood (data fit)

Prior (toward 0)

Posterior (product)

MLE — data's best fit

MAP — posterior peak

Origin (0, 0)

MLE $(\theta_1,\theta_2)$ —

MAP $(\theta_1,\theta_2)$ —

Shrinkage $\lVert\theta_{\text{MAP}}\rVert / \lVert\theta_{\text{MLE}}\rVert$ —

Three things to try. (1) Drag Prior width $\tau$ to the left (narrow, strong prior) with the Gaussian prior: the MAP slides smoothly toward the origin — ℓ₂ shrinks everything a little. (2) Switch to the Laplace prior and narrow $\tau$: the MAP snaps onto an axis, driving $\theta_2$ to exactly zero — that corner of the diamond is ℓ₁ sparsity. (3) Push Amount of data to the right (or drop the noise): the blue likelihood tightens and the MAP marches back onto the MLE — with enough data, the prior stops mattering.

The Payoff

Why ℓ₁ gives sparsity and ℓ₂ does not

Look at the prior contours. The Gaussian prior draws smooth ellipses — round everywhere. When the likelihood ellipse presses against it, they meet at a generic point with both coordinates nonzero: ℓ₂ shrinks weights toward zero but rarely to zero.

The Laplace prior draws diamonds — with sharp corners sitting exactly on the axes. A tilted likelihood ellipse is very likely to first touch the diamond at a corner, and a corner has one coordinate equal to zero. That geometric fact — pointy ℓ₁ ball versus round ℓ₂ ball — is the entire reason lasso produces sparse models and ridge does not.

Takeaway. Regularization strength is prior width: a narrow prior ($\tau\!\downarrow$, big $\lambda$) is a strong belief that weights are near zero; a wide prior ($\tau\!\uparrow$, small $\lambda$) barely constrains them. And the likelihood's grip tightens with data, so $\hat\theta_{\text{MAP}}\to\hat\theta_{\text{MLE}}$ as $N\to\infty$. Frequentist regularized regression and Bayesian MAP estimation are the same computation viewed from two sides.