Perceptual Quality, Pixel by Pixel

Why "MSE was fine, actually" stopped being true

For most of image processing's history, two images were compared by mean squared error (MSE) or its log-scaled cousin PSNR. Compress a photograph; measure the MSE; report a number; ship. The procedure has the virtue of being differentiable, cheap, and uncontroversial.

It also has one catastrophic problem: the human visual system does not perceive error pixel-by-pixel. A constant brightness offset of 5/255 across an image is essentially invisible — yet contributes the same MSE as changing every tenth pixel by 50/255, which is glaringly obvious. Same MSE, two visual planets apart.

The reaction was a sequence of metrics that try to encode increasingly accurate models of what humans actually see: local structure (SSIM), multi-scale structure (MS-SSIM), mid-level perceptual features (LPIPS), and — for generative models where there is no reference — distribution-level metrics (FID, KID).

PSNR — MSE in decibels

Given a reference image $x$ and a distorted copy $y$, with each pixel in $[0, L]$ (typically $L = 255$ or $L = 1$):

PSNR is just the negative log of MSE, normalised by the dynamic range. The decibel scale compresses several orders of magnitude into a single number you can compare across codecs. As a back-of-envelope:

• $> 40$ dB — visually identical for most content.
• $30$–$40$ dB — small artefacts, often acceptable.
• $20$–$30$ dB — visible degradation.
• $< 20$ dB — heavily distorted.

These thresholds are indicative, not perceptual law. The same 30 dB image can look pristine if the error is uniform brightness, and like garbage if it is salt-and-pepper noise. The interactive cards below make this concrete.

PSNR's three honest virtues: it is convex in $y$, it admits a closed-form gradient, and it is what your codec publication will be compared on. Its three honest sins: uniform per-pixel weighting, blindness to spatial structure, and blindness to anything the human visual system actually exploits.

SSIM — luminance, contrast, structure

Wang et al. (2004) made a small, brilliant observation: perceived similarity can be decomposed into three local comparisons — does brightness match, does contrast match, does the pattern match? Run each on a small sliding window, multiply, average.

On a window with local means $\mu_x, \mu_y$, local variances $\sigma_x^2, \sigma_y^2$, and local covariance $\sigma_{xy}$, define

The product $l \cdot c \cdot s$ collapses to the iconic single-line SSIM formula

with stabilisers $C_1 = (0.01 L)^2$, $C_2 = (0.03 L)^2$. Average over all windows to get the image-level SSIM. Range: $[-1, 1]$; identical images give $1$.

What this buys you:

• Luminance-invariant under shifts. If you add a constant to every pixel, $\mu_x$ and $\mu_y$ both shift; $l$ stays near 1. PSNR collapses.
• Contrast-invariant under scaling. If you multiply by a constant, $c$ stays near 1.
• Structure is normalised cross-correlation. The $s$ term is exactly $\sigma_{xy} / (\sigma_x \sigma_y)$ — Pearson correlation in a window. This is the part that actually responds to where the edges and textures sit.

MS-SSIM — and why one window isn't enough

A single 11×11 window has a single spatial frequency band of sensitivity. But human vision is multi-scale: an identical SSIM at 8×8 can correspond to two perceptually different images if one preserves coarse structure and the other preserves fine. Multi-Scale SSIM (Wang et al., 2003) computes SSIM at $M$ downsampled levels and combines them as a weighted product:

The weights $\alpha, \beta_j, \gamma_j$ were fitted on human-rated distortions. The headline benefit: MS-SSIM tracks subjective quality better than single-scale SSIM for blurring and compression, and it became the standard for image-compression evaluation.

Once you see this, you start asking the bigger question: if multi-scale structure is the right thing to compare, why define structure by hand? Could we let a network learn it?

LPIPS — distance in a learned feature space

LPIPS (Zhang et al., 2018, "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric") replaces hand-crafted structure with the activations of a pretrained image classifier. Run both images through, say, AlexNet or VGG-16; extract feature maps $\phi^{(\ell)}(x)$ and $\phi^{(\ell)}(y)$ at several layers; compute a per-channel learnt weighting and take the L2 distance.

The weights $w_\ell$ are trained on the BAPPS dataset: human raters were given two distorted versions of the same reference and asked which looked more like the original. LPIPS is the linear combination of per-layer feature distances that best agrees with those 2AFC choices.

Three things this gets you that SSIM does not:

• Texture invariance. Two patches of the same grass texture have small LPIPS even when they have zero pixel overlap — the deep features encode what the texture is, not where each blade is.
• Robustness to small geometric shifts. CNN feature maps pool spatially, so a one-pixel shift barely moves LPIPS. SSIM and PSNR both spike.
• Sensitivity to semantic content. A cat with a wrong-coloured eye is "close to a cat" in feature space; in pixel space it's whatever the eye patch's MSE happens to be.

Live: distort a reference, watch the metrics

A small synthetic test image — gradient sky, a circle, a checkered patch, and a striped band. The left panel is the unmodified reference. The right panel is the same image after the distortion you choose. Watch how each metric responds.

The "Perceptual proxy" column is a hand-rolled stand-in for LPIPS: a multi-scale gradient-feature L2 distance. It is not LPIPS — a real LPIPS needs a pretrained CNN — but it captures the same spirit (distance in a feature space that weights edges and textures over flat regions).

Things worth trying:

• Crank brightness shift to a value that destroys PSNR — and watch SSIM barely move.
• Add a small blur. SSIM and the perceptual proxy drop hard; PSNR drops less than you'd expect.
• Heavy JPEG block at strength ≈ 60 produces unmistakeable 8×8 boxiness; the perceptual proxy spikes far more than PSNR.
• Compare noise σ = 0.08 with blur σ = 0.8. Both can land near 25 dB but feel very different.

The classic trap: same PSNR, different quality

This is the figure that launched a thousand SSIM citations (essentially Wang & Bovik's original 2002 demo). Below are five distorted versions of the same reference, each with their distortion strength tuned so that PSNR is approximately equal — yet they look obviously different, and SSIM and the perceptual proxy reflect that.

The brightness-shift card has the same MSE as the others — but to the eye it is almost indistinguishable from the reference, and SSIM reflects that. The block-quantised card looks the worst; its SSIM and perceptual proxy are the worst by a margin. PSNR cannot tell them apart.

When there is no reference: FID, KID, IS

PSNR, SSIM and LPIPS are full-reference metrics — they need a clean ground-truth image. For generative models (GANs, diffusion, VAEs) there is no reference for an individual sample; what we want is "did this model produce samples that collectively match the data distribution?". For that we go to distributional metrics.

• Inception Score (IS). Salimans et al., 2016. Push samples through Inception-v3, get class probabilities $p(y \mid x)$. Reward "confident classifications" and "diverse marginal" via a KL term. Problem: it ignores the real data distribution entirely.
• FID (Fréchet Inception Distance). Heusel et al., 2017. Take Inception-v3 pool-3 features for $N$ real samples and $N$ generated samples. Fit a Gaussian to each. Compute the Fréchet distance between them: $\|\mu_r - \mu_g\|^2 + \mathrm{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})$. Lower is better; identical distributions give 0.
• KID (Kernel Inception Distance). Bińkowski et al., 2018. Replace the Gaussian assumption with the unbiased MMD using a polynomial kernel on Inception features. Less biased on small sample sizes (FID is biased and the bias is dataset-size dependent; KID has a tractable unbiased estimator).
• Precision and Recall. Sajjadi et al., 2018; Kynkäänniemi et al., 2019. Decompose distribution match into "do generated samples lie in the data manifold" (precision) and "does the data manifold get covered" (recall). Diagnoses mode collapse that FID can hide.

When to use what

Sharp edges and lab-aligned notes

• SSIM on dynamic-range mismatched data is meaningless. If your reference is [0, 1] float and your prediction is $[0, 255]$ uint8, SSIM will silently rescale and you'll get junk numbers. Always normalise both inputs to the same range, and pass the dynamic-range $L$ explicitly.
• LPIPS on grayscale needs a backbone choice. VGG-16 and AlexNet both expect 3-channel inputs. Standard practice: replicate the gray channel three times. For thermal, this means LPIPS is reading ImageNet-shaped features into a thermal image, which is not what the metric was validated on. Treat LPIPS-on-thermal as indicative, never authoritative.
• SSIM with sliding window can be expensive. For very large images use the integral-image trick (compute moments via summed-area tables) — $O(\text{pixels})$ instead of $O(\text{pixels} \cdot \text{window}^2)$.
• FID needs ≥ 10k samples. Below 10k it's dominated by the small-sample bias term. KID is a better choice in the few-thousand regime.
• Beware metric optimisation. If you train a model with LPIPS as the loss, the model will learn to game the VGG features that LPIPS reads. The resulting samples can be very low LPIPS and quite ugly. This is the perceptual analogue of Goodhart's law and shows up routinely in super-resolution work — for super-resolution, GAN-based losses combined with LPIPS generally beat pure-LPIPS optimisation.
• For our lab (thermal / VLM / super-resolution work): report PSNR + SSIM (with $L = 1$ on normalised inputs) as the workhorses, LPIPS as a secondary signal, and FID only when comparing distributions of generated thermal frames. The "same PSNR, different SSIM" gallery above is the trap to keep in mind when ApneaEye / JoulesEye reconstructions are evaluated against ground-truth thermal frames.

Reading list

• Wang, Bovik, Sheikh, Simoncelli (2004) — Image Quality Assessment: From Error Visibility to Structural Similarity. The SSIM paper. Every paragraph still earns its keep.
• Wang, Simoncelli, Bovik (2003) — Multi-scale Structural Similarity for Image Quality Assessment. MS-SSIM weights and method.
• Zhang, Isola, Efros, Shechtman, Wang (2018) — The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. The LPIPS paper, plus the BAPPS dataset.
• Heusel, Ramsauer, Unterthiner, Nessler, Hochreiter (2017) — GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Introduces FID.
• Bińkowski, Sutherland, Arbel, Gretton (2018) — Demystifying MMD GANs. Introduces KID.
• Kynkäänniemi, Karras, Laine, Lehtinen, Aila (2019) — Improved Precision and Recall Metric for Assessing Generative Models.
• Mantiuk et al. (2011) — HDR-VDP-2. A more sophisticated, HDR-aware perceptual metric — useful when SSIM/LPIPS feel too blunt.
• Sara, Akter, Uddin (2019) — Image Quality Assessment through FSIM, SSIM, MSE and PSNR — a Comparative Study. Good overview when you want a single survey to point students at.