Domain Shift in Vision, In Depth

Prelude

Where the gap opens up

Test accuracy on the test set was 92%. Production accuracy is 67%. Why? In every paper this is called "domain shift" or "distribution shift", but those phrases hide six very different problems. You have to know which one you have before you can fix it.

Type	$p_{\text{src}}$ vs $p_{\text{tgt}}$	Vision example	Symptom
Covariate shift	$p(x)$ differs; $p(y\mid x)$ same	Same kilns, different sensor / season / illumination	Predictor still right where it triggers, but rarely triggers
Label shift	$p(y)$ differs	10× more rare class in target	Confusion matrix tilts toward the new majority
Concept drift	$p(y\mid x)$ changes over time	Definition of "fire" expanded after policy change	Calibration silently breaks even with same accuracy
Sub-population shift	$p(\text{group})$ differs	Trained on one camera; deployed across many	Worst-group accuracy collapses
Open-set shift	New classes at test	New land-use type appears	Confidently wrong predictions on never-seen classes
Adversarial shift	Targeted perturbation of $x$	Stickers, sensor jamming, weather attacks	Catastrophic, point-wise

The slogan. The right tool depends on the type of shift, the access you have to target data (none / unlabelled / few-shot / fully labelled), and your training budget. The next two sections map techniques to each combination.

Step 1

The 2-D playground (live)

Drift the target relative to source; watch the source-only classifier collapse on target. Tick CORAL to feature-align; watch most of the gap close. Energy heatmap and OOD ROC show how confidently you can flag truly-OOD points.

drift 1.5 apply CORAL alignment source acc — target acc — OOD AUC —

Source (●) + target (×) + classifier

Energy score (lighter = more OOD)

OOD ROC (target vs distant noise)

Step 2

Ten vision-specific adaptation techniques, walked through

What follows is a working summary of the techniques you'll meet in the literature and how to actually use each one in a vision pipeline. Order: cheap → expensive.

2.1 BatchNorm-stats adaptation (AdaBN)

The cheapest fix in the book. Take your trained model; re-compute the running mean / variance of every BatchNorm layer using a few hundred unlabelled target images; freeze the affine $\gamma, \beta$. Done in 30 seconds, no back-prop. Often closes 30-60% of the gap when shift is mostly low-level (illumination, sensor noise).

Caveats. Doesn't help if shift is semantic (new objects, new classes). Doesn't help with plain LayerNorm / GroupNorm models — those have no running stats to re-fit. For ViTs, the analog is "recalibrate the LayerNorm affine on a held-out batch of target."

2.2 CORAL — second-order moment matching

Whiten source features by $C_S^{-1/2}$, re-colour by $C_T^{1/2}$, train the head on the matched source. The live demo above does exactly this. Strong baseline for tabular and feature-level alignment; less effective on raw pixels.

2.3 MMD — kernel maximum mean discrepancy

Add a loss term that pushes the kernel-mean of source features close to the kernel-mean of target features. Implemented as the MK-MMD penalty in DAN (Long et al., 2015) and DDC. Useful when you want to align distributions beyond second-order statistics.

2.4 DANN — domain-adversarial training

Add a small domain classifier head that tries to tell source from target. Train the feature extractor to fool it via gradient reversal (Ganin & Lempitsky, 2015). Features become domain-invariant and the downstream task head still works. DANN is what every subsequent adversarial DA method (CDAN, MDD, ADDA) refines.

Implementation tip. The gradient-reversal layer is just the identity in forward and a sign-flipping multiplier $-\lambda$ in backward. Schedule $\lambda$ from 0 to 1 over training; otherwise the adversarial signal is too strong early and breaks the task head.

2.5 MixStyle — domain-augmenting style mix

Inside the network, mix the channel-wise mean and std of two examples from different (or random) source domains (Zhou et al., 2021). Cheap data augmentation that simulates unseen styles. Strong for cross-camera and cross-illumination shifts.

2.6 IRM & group-DRO — robust ERM

When you have multiple labelled source domains, Invariant Risk Minimisation (Arjovsky et al., 2019) and Group Distributionally Robust Optimisation (Sagawa et al., 2020) train a model to do well on the worst environment, not the average. Useful when the test domain is "unseen but related."

2.7 Test-time adaptation — TENT & SHOT

Adapt at deployment with no source data. TENT (Wang et al., 2021) updates the BatchNorm affine parameters by minimising prediction entropy on the test batch. SHOT (Liang et al., 2020) goes further: source-free DA where you only ship the source-trained model, then adapt features on target with self-training pseudo-labels. Both work on small streaming batches; both can break if the batch is class-imbalanced or includes too much OOD — entropy minimisation collapses to a single class.

2.8 Pseudo-labelling + self-training

The workhorse for semi-supervised DA. Use the source model to produce labels on confident target samples, retrain the model on (source labelled ∪ target pseudo-labelled). Iterate. Combine with consistency regularisation (FixMatch, MixMatch) for stronger results.

2.9 Prompt / adapter / LoRA-based DA on big backbones

For large pretrained backbones (CLIP, DINOv2, SAM), a tiny domain-specific prompt or LoRA adapter often closes the gap with a fraction of the parameters of full DA training. Visual Prompt Tuning, CoOp / CoCoOp, AdapterHub. Especially attractive when the source data is private or unavailable.

2.10 Diffusion-based domain translation

When source and target are visually different (sim-to-real, rainy → sunny, season change), train a conditional diffusion model that translates a target image into a source-flavoured one (or vice versa) before inference. CycleGAN was the GAN-era version; today ControlNet / IP-Adapter / SDXL-Turbo translations are the practical stack.

Step 3

The decision tree — pick the technique in 30 seconds

Q1: Do you have labelled target? If any labelled target → fine-tune a head + LoRA on target; you don't need DA. Stop.
Q2: Unlabelled target only, can re-train? Try AdaBN first (free). If the model uses BatchNorm and the shift is illumination / sensor / weather, this often closes most of the gap. If insufficient, add CORAL or MMD on features. If still insufficient, train DANN.
Q3: No target data at training time, just at test? Test-time adaptation: TENT for BN models; SHOT for source-free; otherwise rely on a robustly-trained backbone (DINOv2 features + small head).
Q4: Multiple labelled source domains, no target? Domain generalisation: IRM / group-DRO / MixStyle augmentation. Or pretrain on all sources and rely on regularisation.
Q5: Big visual gap (sim → real, day → night)? Diffusion-based domain translation; or aggressive style augmentation; or both.
Q6: Privacy / no source access? Source-free DA (SHOT, AdaContrast); ship only the model and adapt on target with pseudo-labels + entropy minimisation.

Default recipe for vision in 2026. (1) DINOv2 backbone. (2) Linear / LoRA head on source. (3) AdaBN if BN is in the architecture (it usually isn't for ViT). (4) TENT at test time. (5) Wrap predictions in conformal prediction for guaranteed coverage. This stack handles most shifts you'll see; reach for DANN / CORAL / MMD only when the backbone is small or the gap is huge.

Step 4

OOD detection — knowing when not to predict

Adaptation closes the gap when source and target overlap. Truly out-of-distribution inputs need a separate detector and a refuse-to-predict path. Five detectors worth knowing:

Max softmax probability (MSP). Cheapest, surprisingly hard to beat at scale. Threshold on $\max_c p(c\mid x)$.
Energy score (above). Strict improvement on MSP; same compute. The default in 2024+.
Mahalanobis on penultimate features. Fit a Gaussian per class on training features; score new inputs by min-class Mahalanobis distance. Good when the backbone gives clean clusters.
ViM, KNN-OOD. Match against the penultimate feature bank with virtual logits or a k-NN distance. Strong on large vision benchmarks (OpenOOD).
Outlier exposure. Add a tiny dataset of "obvious OOD" images during training with a uniform target; the model learns to be uncertain off-distribution. Cheap and reliable.

Step 5

Worked example — sim-to-real for an object detector

Concrete pipeline for a common vision DA setup: train an object detector in simulation, deploy on real photos. Standard recipe in robotics / autonomy / agritech.

Pretrain backbone on real photos (free). DINOv2 or a CLIP-flavoured backbone. Don't train from scratch on simulated data — you'll bake in the sim aesthetic.
Train detector on simulation (labelled). Standard DETR / RT-DETR / YOLOv10 head; freeze backbone for the first 5 epochs to avoid corrupting features.
Add aggressive style augmentation. MixStyle in feature space + colour-jitter + RandAugment on the simulator output. Simulator weather diversity helps too.
Domain-translation pre-step (optional). Run a sim-to-real CycleGAN / ControlNet over each batch during training. Reduces the visual gap by a lot at the cost of more compute per batch.
Real-data fine-tune (if any labelled real available). Even 100 labelled real images + existing 100k sim images, fine-tuned end-to-end with 10× lower lr, beats sim-only by a wide margin.
Test-time adaptation. TENT on detection heads + a small EMA-of-self pseudo-labelling loop.
OOD gate. An energy-score threshold: don't emit boxes if the patch energy is below threshold. Always wraps a deployed detector.
Conformal calibration. Conformalise the detector's confidence so that "0.85" actually means 85% precision; see the conformal article.

Step 6

Worked example — multi-site medical / satellite shift

When the "domains" are sites (hospitals, sensor stations, satellite passes), the problem is domain generalisation, not single-target adaptation. Recipe:

Hold-one-site-out evaluation. Always. Random splits leak by site and over-state generalisation by 10-30 points.
Per-site stratified sampling during training. Equal weight per site, not per image. Keeps any site from dominating the gradient.
Group-DRO loss. Optimise for the worst-site loss. Costs a small accuracy hit on average, wins on worst-case.
Site-conditioning at training, not at test. Encode site as an embedding fed to the backbone during training; at test drop it and rely on generalisation. Helps disentangle site effects from class effects.
Hierarchical Bayes per site. See the partial-pooling section; each site has its own posterior, pulled toward a global prior. Especially relevant for small data per site.
OOD detector per site. Train an OOD detector that flags inputs that look unlike any seen site.

Step 7

The bugs and gotchas

"BN in eval mode is fine on target." No. Eval mode locks running mean/var to source values; TENT / AdaBN exist precisely because this loses points.
"DANN converged" but accuracy is bad. Adversarial DA can find solutions where domain-invariant features are also class-invariant. Always evaluate on target accuracy, not the adversarial loss.
Random target preview during development. Tempting to peek at target during model selection; it biases your hyperparameter choice toward the test set. Use unsupervised selection criteria (e.g., self-confidence on target) when you don't have target labels.
"My model just does worse with augmentation." Style-augmentation and weather-aug help shifts they simulate. If your shift is sensor-band reordering or spectral, RGB-augmentation won't touch it.
OOD detector picked on the wrong score. Energy > MSP > Mahalanobis on most benchmarks, but OpenOOD shows the right pick depends on backbone and shift type. Test all three; pick by AUROC on a synthetic OOD set.
Simulation overfit. Detectors trained only on sim learn to recognise the simulator's renderer. Real-world deployment fails. Domain randomisation or sim-to-real translation is non-negotiable.
Spectral / multi-modal shift. CORAL on RGB doesn't help when target is multi-spectral. Pretrain on an EO foundation model and adapt at the feature level instead.

Final takeaway. The hard part of domain shift in vision is rarely choosing a loss function — it's diagnosing which kind of shift you have and matching the technique to it. The decision tree in Step 3 plus an honest hold-one-site-out evaluation in Step 6 will save you a year of debugging.