Interactive Explainer

Softmax & Cross-Entropy

Classification networks output raw scores. Softmax turns them into a probability distribution; cross-entropy scores how surprised the model is by the true answer; and — the beautiful part — the gradient that trains the network turns out to be exactly predicted minus target, $p - y$. Slide the logits below and watch all three move together.

~9 min Deep Learning · Loss Functions · Classification

Prelude

Three quantities, one picture

For a $K$-class problem the network emits logits $z_1,\dots,z_K$. Softmax with temperature $T$ maps them to probabilities, cross-entropy reads off the loss for a one-hot target $y$, and the gradient with respect to the logits is startlingly clean:

p_k = \frac{e^{z_k/T}}{\sum_j e^{z_j/T}}, \qquad \mathcal{L} = -\log p_{y}, \qquad \frac{\partial \mathcal{L}}{\partial z_k} = p_k - y_k .

That last identity is why softmax + cross-entropy is the default pairing in deep learning: no messy chain rule survives, just “how wrong is each class, signed.” The widget below is built entirely around these three formulas.

The Lab

Logits in, loss and gradient out

Three classes. Set the logits, choose which class is actually correct, and optionally sharpen or flatten with temperature. The left panel is the softmax distribution (the true class outlined); the right panel is the gradient $p - y$, the signed force each logit feels during backprop.

Logit $z_1$ = 3.00

Logit $z_2$ = 0.50

Logit $z_3$ = -1.00

True class $y$

Temperature $T$ = 1.00

Softmax probabilities $p_k$. The outlined bar is the true class; its height is $p_{\text{true}}$.

Gradient $p_k - y_k$. Bars above zero get pushed down; the bar below zero (true class) gets pushed up.

Cross-entropy loss $\;\mathcal{L} = -\log p_{\text{true}}$ —

Probability of true class —

Model's top pick —

Gradient norm $\lVert p-y\rVert$ —

Click through the four presets. Watch the big loss number. Correct & confident is nearly zero. Wrong & confident is huge — being sure and wrong is the single most expensive thing a classifier can do, because $-\log p_{\text{true}} \to \infty$ as $p_{\text{true}} \to 0$. The gradient bar on the true class also grows toward $-1$: the strongest possible push to fix the mistake.

The Payoff

Why the gradient is just $p - y$

Cross-entropy looks like it should have an ugly derivative — there is a $\log$, and a $\sum$ inside the softmax. But the $\log$ and the $\exp$ cancel almost perfectly. For the true class $y$:

\frac{\partial}{\partial z_k}\Big(-\log p_y\Big) = p_k - \mathbb{1}[k = y] = p_k - y_k .

For a non-true class, $y_k = 0$, so the gradient is $+p_k$ — push that logit down in proportion to how much probability it wrongly claimed.
For the true class, $y_k = 1$, so the gradient is $p_{\text{true}} - 1 \le 0$ — push that logit up, hardest when $p_{\text{true}}$ is small.
The gradients sum to zero: probability that leaves the wrong classes flows into the right one. Softmax conserves it.

Practical note. Because the gradient is $p - y$, frameworks fuse softmax and cross-entropy into a single op (PyTorch's CrossEntropyLoss takes raw logits, not probabilities). Applying softmax yourself and then a separate cross-entropy double-counts and gives wrong, unstable gradients. Feed logits; let the loss do the softmax.