Saliency, Side by Side

The 'why' question

A classifier outputs "tumor present, p = 0.92" on a scan. The right next question is why. Saliency methods produce a per-pixel "importance" map answering "which pixels of the input most influenced this prediction?". They are useful, sometimes deeply misleading, and almost always worth running before publishing a model or deploying it.

The four methods

• Vanilla gradient. $\partial f_c / \partial x$. Cheapest, noisiest. Saturates on ReLUs (zero gradient where the unit was off).
• SmoothGrad. Average vanilla gradients over $n$ noisy copies of the image. Smoother, more stable; still sensitive to the model.
• Integrated Gradients. Integrate gradients along a path from a baseline (often zeros) to the input. Satisfies completeness: pixel attributions sum to (output − baseline-output).
• Grad-CAM. Gradient-weighted activation map at a chosen conv layer. Coarser (one-pixel-per-CNN-cell) but faithful to the network's internal feature space.

For Integrated Gradients, $x^\prime$ is the baseline (often a black image). The integral guarantees completeness: $\sum_i \mathrm{IG}_i = f_c(x) - f_c(x^\prime)$ — every unit of the prediction is attributed to some pixel. No gradient method without an integration path has this property.

Axioms — why we should care about them

Sundararajan, Taly, Yan (2017) introduced an axiomatic frame for evaluating attribution methods. The four key axioms:

• Sensitivity. If $x$ and $x^\prime$ differ on a single feature and give different predictions, that feature must get non-zero attribution. Vanilla gradient fails this because of ReLU saturation; IG satisfies it.
• Implementation invariance. Two functionally-identical networks should yield identical attributions. Vanilla gradient passes; some methods that hold gradient flow at activations (DeepLIFT, LRP) can fail in pathological cases.
• Completeness. Attributions sum to the prediction minus the baseline. Only path-integrated methods (IG, Shapley values) satisfy this.
• Linearity. Attribution of $\alpha f + \beta g$ equals $\alpha$ × attribution of $f$ + $\beta$ × attribution of $g$. Satisfied by all linear-in-gradient methods.

The axioms uniquely pick out the path-integrated gradient when you also require Shapley-style symmetry. That's why Integrated Gradients (and its Shapley relatives, like Expected Gradients) are the academically-correct choice when you can afford them.

Live, on a synthetic image

Below: a hand-crafted small CNN (just 1 conv + global mean + linear head, with manually-chosen weights) classifies a synthetic "thermal" image. Toggle the image and watch the four saliency maps respond. The hot regions in the image are the ground-truth signal; a faithful method should focus its attribution there.

The traps

• Vanilla gradient is noisy. Single-pixel differences scatter; tiny network non-linearities dominate. SmoothGrad and IG are usually more readable.
• IG depends on the baseline. Black image isn't always the right "absence of feature". For thermal, a constant-temperature reference is more honest.
• Grad-CAM is class-discriminative per layer. Pick the wrong layer and you get a uniform blob. Conventionally use the last conv layer.
• None of these is causal. Saliency is "the gradient of $f$ at $x$", not "what would happen if we removed this pixel". For causal attributions look at occlusion, RISE, or perturbation-based methods.
• Adebayo's sanity check. Permute the model's last layer; rerun saliency. If the maps look the same, your saliency is reading the input, not the model.

Beyond gradient-based: perturbation methods

Gradient methods answer "if I changed this pixel infinitesimally, how would the prediction change?". A perturbation method asks the stronger, causal question: "if I removed this pixel entirely, how would the prediction change?". Three families:

• Occlusion (sliding window). Replace each region with a baseline (gray patch, blur) and measure the drop in $f_c$. Slow ($O(\text{pixels} \times \text{forward passes})$) but model-agnostic; works on black-box APIs.
• RISE (Petsiuk et al., 2018). Randomly mask the input many times, weight each mask by the resulting $f_c$, average. A Monte-Carlo perturbation method that's both faster than occlusion and noisier than IG.
• LIME / KernelSHAP. Fit a sparse linear surrogate to the model's behaviour on perturbations near $x$. KernelSHAP additionally enforces Shapley-value properties.
• Causal interpretations. If you want true counterfactual reasoning ("would the prediction be different had this feature been absent?"), look at the concept bottleneck article — intervention on an explicit concept is closer to what people usually mean by "why".

Sanity checks you should always run

• Adebayo's randomisation test. Randomise the model's parameters layer-by-layer. If the saliency map doesn't change, your map is reading the input, not the model — you have a fancy edge detector, not an explanation.
• Data randomisation. Train two models, one with shuffled labels. Saliency on the shuffled-label model should be qualitatively different.
• Sensitivity-N / pointing game. Mask the top-K pixels by attribution; measure prediction drop. Average over a dataset. Compare methods on the same model.
• Pixel-flipping / insertion-deletion AUC. Insert pixels by descending attribution; plot $f_c$ vs fraction inserted; the area-under-curve is a quantitative quality measure (Petsiuk et al., 2018).

Saliency for vision transformers

ViTs don't have convolutional feature maps, so Grad-CAM doesn't apply directly. Three replacements:

• Attention rollout. Multiply attention matrices across layers (with residual). Coarse but parameter-free; see the ViT article.
• Attribution rollout (Chefer et al., 2021). Combine attention with gradients; outperforms vanilla rollout on most benchmarks.
• Token-level IG. Integrate gradients with respect to token embeddings instead of pixels. Native attribution scale = one number per patch.

Reading list

• Simonyan, Vedaldi, Zisserman (2014) — Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. The original.
• Sundararajan, Taly, Yan (2017) — Axiomatic Attribution for Deep Networks. The IG paper.
• Selvaraju et al. (2017) — Grad-CAM.
• Smilkov et al. (2017) — SmoothGrad.
• Adebayo et al. (2018) — Sanity Checks for Saliency Maps.
• Petsiuk et al. (2018) — RISE.
• Chefer, Gur, Wolf (2021) — Transformer Interpretability Beyond Attention Visualization.