Interactive Explainer

From Likelihood to Loss

Where do loss functions come from? Not from thin air. Assume your data's residuals follow some noise distribution $p(r)$, then ask for the parameters that make the data most likely. Maximizing likelihood is minimizing $-\log p(r)$ — and that is your loss. Change the noise model, change the loss. This page keeps three panels in sync so you can watch the loss fall out of the density in real time.

~11 min Deep Learning · Loss Functions · Maximum Likelihood

Prelude

The one identity that runs the whole show

Fit a line $\hat{y}_i = w\,x_i + b$ to data. Each point has a residual $r_i = y_i - \hat{y}_i$. Now model those residuals as random noise drawn from a density $p(r)$. Assuming the points are independent, the likelihood of the whole dataset is the product $\prod_i p(r_i)$. Maximizing it is the same as minimizing the negative log-likelihood:

\underbrace{-\log \prod_i p(r_i)}_{\text{negative log-likelihood}} = \sum_i \underbrace{\big(-\log p(r_i)\big)}_{\text{per-point loss}} .

So the loss function is literally $-\log p(r)$. Every choice you have ever made — MSE, MAE, a Huber loss — is a bet about the shape of the noise. Assume Gaussian noise and you have just derived squared error. Assume Laplace noise and out pops absolute error. Let's see it.

The Lab

Density, loss, and fit — all at once

Pick a noise model. The left panel draws its density $p(r)$; the right panel draws the resulting loss $-\log p(r)$. Below them is a small dataset with a line you control. The colored dots on the density and loss curves are your actual residuals; the vertical stubs on the data panel are the same residuals down on the ground. The total loss is the sum of those per-point heights.

Noise scale $\sigma$ = 1.00

Line slope $w$ = 0.60

Line intercept $b$ = 1.00

Add one outlier

Noise density $p(r)$. Dots mark each residual's likelihood.

Per-residual loss $-\log p(r)$. Dot height = that point's loss.

Data with your fitted line. Each vertical stub is a residual, tinted by how much loss it contributes (green = cheap, red = expensive).

Total loss $\sum_i -\log p(r_i)$ —

Worst single point —

Loss grows like —

Do this. Turn on Add one outlier with the Gaussian model, then hit Fit selected loss. Watch the whole line tilt to chase the one bad point — squared error pays $r^2$, so a far point dominates the sum. Now switch to Laplace or Student-t and snap again (Laplace's fit is the median-like line). The robust losses barely flinch, because their tails grow far more slowly.

The Payoff

Why the tails are the whole story

Look only at the right-hand loss panel and how it behaves for a large residual:

Noise model	Density $p(r)$	Loss $-\log p(r)$	Far-tail behavior
Gaussian	$\propto e^{-r^2/2\sigma^2}$	squared error $\;\tfrac{r^2}{2\sigma^2}$	grows like $r^2$ — outliers dominate
Laplace	$\propto e^{-\|r\|/b}$	absolute error $\;\|r\|/b$	grows like $\|r\|$ — robust
Uniform on $[-a,a]$	$\tfrac{1}{2a}$ inside, $0$ outside	constant inside, $+\infty$ outside	a hard wall — any point outside is forbidden
Student-t ($\nu=3$)	$\propto \big(1+\tfrac{r^2}{\nu\sigma^2}\big)^{-\frac{\nu+1}{2}}$	$\tfrac{\nu+1}{2}\log\!\big(1+\tfrac{r^2}{\nu\sigma^2}\big)$	grows like $\log r^2$ — very robust

That last column is the punchline. A loss that grows like $r^2$ lets a single far point buy an enormous vote; a loss that grows like $|r|$ or $\log r$ caps its influence. “Robust regression” is nothing more than assuming a heavier-tailed noise model.

Takeaway. You do not choose a loss function — you choose a noise model, and the loss is $-\log p(r)$ whether you say so or not. MSE assumes Gaussian noise. MAE assumes Laplace noise. If your data has outliers, that is your model telling you the Gaussian tail is too thin.