Interactive Explainer

A Bayes Classifier, by Hand

If you knew the true distribution of each class, what is the best possible classifier? Bayes' rule answers exactly: form the posterior $p(Y=1\mid x) \propto p(x\mid Y=1)\,p(Y=1)$ and predict the more probable class. Here we build it from two Gaussians and a prior — and watch the optimal decision boundary appear right where the posterior crosses one half.

~10 min Deep Learning · Probability · Classification

Prelude

Likelihood times prior, normalized

Model each class as a generator of data. Class 0 emits points from $p(x\mid Y=0)=\mathcal{N}(\mu_0,\sigma_0^2)$; class 1 from $p(x\mid Y=1)=\mathcal{N}(\mu_1,\sigma_1^2)$. Add a prior belief $\pi = p(Y=1)$ about how common class 1 is. Bayes' rule inverts the generator to tell you, given a point $x$, how probable each class is:

p(Y=1\mid x) = \frac{\overbrace{p(x\mid Y=1)}^{\text{likelihood}}\,\overbrace{\pi}^{\text{prior}}} {p(x\mid Y=1)\,\pi + p(x\mid Y=0)\,(1-\pi)} .

Predict class 1 wherever this posterior exceeds $\tfrac12$. The set of points where it equals exactly $\tfrac12$ is the decision boundary. No classifier can do better on average than this one — it is the Bayes-optimal rule, the yardstick everything else is measured against.

The Lab

Build the classifier

Top panel: the two class-conditional densities (already weighted by their priors, so their areas reflect $\pi$ and $1-\pi$). Bottom panel: the posterior $p(Y=1\mid x)$, with the $0.5$ line and the resulting decision boundary. Drag anywhere on the figure — or use the $x^{*}$ slider — to move the test point.

Mean $\mu_0$ -1.5

Mean $\mu_1$ 1.5

Std $\sigma_0$ 1.00

Std $\sigma_1$ 1.00

Prior $\pi = p(Y{=}1)$ 0.50

Test point $x^{*}$ 0.00

Top: prior-weighted densities $\pi\,p(x|Y{=}1)$ and $(1-\pi)\,p(x|Y{=}0)$. Bottom: posterior $p(Y{=}1|x)$. Vertical orange line is $x^{*}$; dashed lines are decision boundaries.

Prediction at $x^{*}$ —

Posterior $p(Y{=}1\mid x^{*})$ —

Decision boundary —

Try this. With equal variances, drag the prior $\pi$ from $0.5$ down toward $0.1$. The boundary slides toward class 1's mean — if class 1 is rare, you demand stronger evidence (a point deeper into its territory) before predicting it. Then make $\sigma_1$ much larger than $\sigma_0$: the boundary splits into two crossings, and the broad class can win out on both far tails. That curvature is exactly what separates quadratic discriminant analysis from the straight-line logistic boundary.

The Payoff

Where the boundary comes from

Setting the posterior to $\tfrac12$ is the same as setting the two prior-weighted likelihoods equal: $\pi\,p(x|Y{=}1) = (1-\pi)\,p(x|Y{=}0)$. Take logs of the two Gaussians and the boundary condition becomes a quadratic in $x$:

-\frac{(x-\mu_1)^2}{2\sigma_1^2} - \log\sigma_1 + \log\pi = -\frac{(x-\mu_0)^2}{2\sigma_0^2} - \log\sigma_0 + \log(1-\pi).

Equal variances ($\sigma_0=\sigma_1$): the $x^2$ terms cancel, leaving a single linear boundary. This is exactly the decision rule of logistic regression / linear discriminant analysis.
Unequal variances: the $x^2$ terms survive — a genuine quadratic with up to two roots. This is quadratic discriminant analysis.
The prior enters only through $\log\frac{\pi}{1-\pi}$, an additive shift — so changing the prior slides the boundary without changing its shape.

Takeaway. A classifier is just a rule for comparing posteriors. Generative models (like this one) get there by modeling $p(x|y)$ and applying Bayes' rule; discriminative models (logistic regression, neural nets) skip straight to modeling $p(y|x)$. Both are chasing the same Bayes-optimal boundary you are dragging around above.