Vision Pretraining, Without Labels

Prelude

Why we don't train from scratch any more

ImageNet has 1.2M labelled images. A typical applied vision task has, optimistically, a few thousand labelled examples. A ViT-Base trained from scratch on that would overfit catastrophically. Self-supervised pretraining on the much larger pool of unlabelled images (millions, easy to scrape; billions, with a bit of ingenuity) gets you a backbone that needs only a thin fine-tune on labels.

Every recipe here invents a pretext task: a supervised-looking objective whose labels come free from the image itself. Two big families dominate. Contrastive methods (SimCLR, MoCo, CLIP) make representations of two views of one image agree while disagreeing with other images. Generative methods (MAE, BEiT) hide part of the image and reconstruct it. We'll build a live demo of each.

The slogan. Spend your unlabelled-data calories on a self-supervised pretext task. Save labelling budget for the part that actually needs labels (the head, sometimes a thin LoRA through the backbone).

Step 1

Contrastive learning & the InfoNCE loss

Take an image, make two randomly augmented views of it (crop, flip, colour-jitter, blur). Push both through an encoder to get embeddings $z_i$ and $z_j$. These two views came from the same image, so they form a positive pair and their embeddings should be close. Every other image in the batch is a negative, and should sit far away. The InfoNCE loss makes that a softmax classification — "which of the $2N$ views is my positive?":

\mathcal{L}_{i} = -\log \frac{\exp\!\big(\operatorname{sim}(z_i, z_j)/\tau\big)} {\sum_{k=1}^{2N}\mathbb{1}[k\neq i]\,\exp\!\big(\operatorname{sim}(z_i, z_k)/\tau\big)}

$\operatorname{sim}(a,b)=\tfrac{a\cdot b}{\lVert a\rVert\,\lVert b\rVert}$ is cosine similarity and $\tau$ is a temperature (smaller = harder negatives, sharper contrast). The numerator pulls the positive pair together; the denominator pushes against every negative in the batch — which is why SimCLR needs large batches (~4k–8k) to see enough negatives, and why MoCo swaps the batch negatives for a momentum-updated queue.

crop + noise strength 0.35

colour-jitter strength 0.65

temperature $\tau$ 0.20

anchor image

view A (positive)

view B (positive)

Embedding space. Each dot is one view's 2-D embedding; the orange line joins the two anchor views. Direction encodes orientation, distance-from-centre encodes how confident the encoder is.

Cosine-similarity matrix over all $2N=8$ views (4 images × 2 views). Orange squares are the positive pairs; InfoNCE wants them to be the brightest entry in their row.

Positive similarity —

Mean negative sim —

InfoNCE loss (anchor) —

Positive is nearest? —

Two knobs, two lessons. Crank colour-jitter to the max: the two views look wildly different in colour, yet the positive similarity barely dips and the positive pair stays the brightest cell in its row. The encoder here is colour-blind by construction — that is the invariance InfoNCE is training in. Now crank crop + noise: augment too aggressively and the views lose their shared structure, the positive dot drifts toward the centre, similarity collapses, and eventually a negative wins — the loss spikes. Good augmentations are strong but structure-preserving.

Augmentations are the whole inductive bias: the network is forced to produce the same embedding for both views, so it learns features invariant to whatever you augmented over. Random crop + colour jitter + Gaussian blur is the standard recipe; dropping any one costs 5–10 ImageNet linear-probe points.

Step 2

The MAE recipe — mask and reconstruct

Masked autoencoders (He et al., 2022) are the simplest generative recipe that works at scale. Divide the image into patches, randomly mask a large fraction (75% is the headline number), feed only the visible patches into a ViT encoder, and ask a small decoder to reconstruct the masked patches. The loss is mean-squared error in pixel space, computed only on the masked patches:

\mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|} \sum_{p\,\in\,\mathcal{M}} \big\lVert \hat{x}_p - x_p \big\rVert^2, \qquad \mathcal{M} = \text{masked patches}

The asymmetry — an encoder that sees only 25% of the patches, a decoder that sees the full set with placeholders for the masked ones — is what makes this fast. Pretraining on ImageNet-1k takes under a day on 8 GPUs. Drag the mask ratio and watch reconstruction quality degrade as fewer patches survive:

mask ratio 0.75

Original

Masked input (encoder sees this)

Reconstruction (decoder output)

—

What the demo can and can't show. We don't run a real ViT in your browser; the 'reconstruction' is a smart interpolation from the visible patches (roughly what an MAE decoder converges to on a smooth scene). Real MAE reconstructions on natural images preserve global structure but blur fine texture — the same qualitative behaviour you see here as you push the mask ratio up.

Step 3

DINO & self-distillation without labels

DINO (Caron et al., 2021) and DINOv2 (Oquab et al., 2023) are the most-used non-contrastive recipe. A student network must match the predictions of a teacher network on a different view of the same image; the teacher is an exponential moving average of the student:

\mathcal{L}_{\text{DINO}} = -\sum_x p_t(x)\,\log p_s(x), \quad p_t = \operatorname{softmax}\!\Big(\tfrac{f_t(x)-c}{\tau_t}\Big), \quad p_s = \operatorname{softmax}\!\Big(\tfrac{f_s(x)}{\tau_s}\Big)

The trick is preventing collapse — both networks predicting a constant. Two stabilisers:

Centering. Subtract a running mean $c$ from the teacher's output before softmax. Removes the easy "always predict the same vector" attractor.
Sharpening. Teacher temperature $\tau_t \ll \tau_s$. The teacher's distribution is sharper than the student's, forcing the student to commit to a confident prediction.

DINOv2 added Sinkhorn-Knopp centering, iBOT-style patch-level loss (the student must also match the teacher on masked patches), and curated pretraining data. The resulting features are spatially well-organised, which is why DINOv2 is the current default for any downstream segmentation or detection task.

Step 4

JEPA — predict features, not pixels

MAE spends capacity predicting pixels: the loss in pixel space punishes you for getting fine textures wrong, which doesn't help downstream semantics. Joint Embedding Predictive Architectures (I-JEPA, V-JEPA; LeCun et al., 2022–24) sidestep this by predicting in feature space:

\mathcal{L}_{\text{JEPA}} = \big\lVert\, g\!\big(f(x_{\text{context}}),\, \ell_{\text{target}}\big) - \operatorname{sg}\!\big[f(x_{\text{target}})\big] \,\big\rVert^2

Both context and target pass through the (same) encoder $f$. A small predictor $g$ predicts the feature embedding of the target from the context's embedding plus a target location code $\ell_{\text{target}}$; $\operatorname{sg}[\cdot]$ is a stop-gradient. By skipping pixel prediction, JEPA spends its parameters on representations useful for downstream tasks, and is increasingly competitive on linear-probe benchmarks at a fraction of MAE's compute.

Step 5

The four families of SSL recipes

Family	Pretext	Examples	What it learns well	What it struggles with
Contrastive	Pull augmentations of same image together; push others apart	SimCLR, MoCo, CLIP	Discriminative features, transfer to classification	Needs huge batches; weak on dense tasks
Distillation	Match a teacher (often EMA of student) on different views	DINO, DINOv2, iBOT	Strong dense features; SOTA for downstream segmentation	Stable training is delicate (centering, sharpening)
Generative	Reconstruct masked pixels / tokens	MAE, SimMIM, BEiT	Sample efficiency; works on any modality	Linear-probe accuracy lower than DINO/CLIP
Predictive	Predict features of masked regions, not pixels	I-JEPA, V-JEPA, V-JEPA 2	Very efficient; emphasis on semantic features	Newer; tooling and recipes still maturing

A useful rule of thumb: for classification-flavoured tasks and small labelled fine-tune sets, prefer DINOv2 or CLIP. For dense prediction, segmentation, or any task where spatial detail matters, prefer DINOv2 or MAE. For unusual modalities (thermal, satellite, hyperspectral) MAE is the easiest to retrofit.

Step 6

Why this matters: linear-probe efficiency

The cleanest way to evaluate pretraining is to freeze the backbone and train a single linear head on a small labelled set. The slope of that "labelled-data vs accuracy" curve is the SSL value. Below: a synthetic curve contrasting random init with SSL pretrained as you grow the labelled fraction.

Synthetic illustration. Real ImageNet linear-probe numbers (Caron et al., 2021): DINO ~78% with 1% labels vs ~10% from random init.

Step 7

The augmentation-as-prior view

The single biggest knob in any non-generative SSL recipe is the augmentation set, for the reason the contrastive lab above made tangible: the loss forces the same representation for two augmented views, so the network learns features invariant to whatever you augmented over. Whatever you don't augment is what the model is free to latch onto.

Random crop. Locality + scale invariance. Removing it tanks ImageNet linear probe by ~25 points.
Colour jitter. Colour-invariance. Crucial for natural images where colour is unreliable.
Gaussian blur. Texture-invariance. Helps especially when the target task cares about shape, not texture.
Solarisation / random greyscale. Used in DINO; further weakens colour as a shortcut.
What to not augment. If your downstream task cares about a property (rotation in medical scans, exact pixel intensity in remote sensing), don't augment it away — the model will learn to ignore exactly the thing you need.

The lesson. Your augmentation set is your inductive bias. Choose with care; revisit it for every new modality.

Step 8

Practical recipes

Custom-modality pretraining. If your domain (thermal, medical, satellite, hyperspectral, microscopy) is unlike ImageNet, an off-the-shelf backbone helps less. Pretrain MAE on tens to hundreds of thousands of in-domain unlabelled images; fine-tune on the few hundred labels you actually have.
Continual pretraining of a public backbone. Cheaper: start from DINOv2 or CLIP weights, run a short MAE-style continual pretrain on your in-domain data, then fine-tune. You get both the big-pretrain generality and the modality alignment.
Linear probe before fine-tune. Always evaluate the frozen backbone with a single linear head first. If the linear probe is bad, fine-tuning rarely saves it.
Vision-language alignment. CLIP-style training on (image, caption) pairs gives you zero-shot classification and free-text retrieval. The captions don't need to be high quality — Web noise is enough.
Don't reinvent. Public DINOv2, CLIP, and SAM backbones cover most "I need a strong vision encoder" needs in 2026. Pretrain from scratch only when no public backbone lives in your domain.

Final takeaway. The hardest part of applied vision isn't training; it's labelling. SSL pretraining lets the unlabelled mountain do the heavy lifting so your annotators only touch the part that fundamentally needs human judgement.

Coda

Reading list

Chen, Kornblith, Norouzi, Hinton (2020) — SimCLR. The contrastive lab above.
He et al. (2020) — MoCo. Momentum queue of negatives.
Caron et al. (2021) — DINO.
He, Chen, Xie, Li, Dollár, Girshick (2022) — Masked Autoencoders Are Scalable Vision Learners (MAE).
Assran et al. (2023) — I-JEPA.
Oquab et al. (2023) — DINOv2. The currently dominant SSL backbone.
Bardes et al. (2024) — V-JEPA. Video extension.