Vision-Language Models, End to End

The architecture in three boxes

Every modern VLM — LLaVA, Qwen-VL, Pixtral, GPT-4o, Claude 3.5 with vision — is the same shape:

• Vision encoder. Usually a CLIP- or DINOv2-pretrained ViT. Takes an image, outputs a sequence of patch embeddings (e.g. 256 tokens of 1024-d for a 224×224 image at patch 14).
• Projector. A small MLP (sometimes a cross-attention) that maps the vision embeddings into the LLM's hidden dimension and (often) reduces the token count via pooling or Q-Former-style learned queries.
• Language model. A pretrained decoder LLM (Llama, Qwen, Mistral, GPT-OSS). Receives projected image tokens prepended to the text token stream and generates an output autoregressively.

Stage 1 — Contrastive image-text alignment (CLIP)

Before a VLM can be assembled, the vision encoder usually needs to be aligned with text. The standard recipe is contrastive: train an image encoder $f_I$ and a text encoder $f_T$ so that matched (image, caption) pairs are near in cosine similarity, and unmatched pairs are far.

$\tau$ is a learnable temperature. The loss is symmetric (image-to-text + text-to-image cross-entropy, then averaged). With 400M+ image-text pairs scraped from the web, this learns a remarkably general visual vocabulary — CLIP can zero-shot classify ImageNet at 76% top-1 without ever seeing a labelled ImageNet image during training.

A mini-CLIP, live

Below: 4 toy "image" types (coloured shapes) and 4 textual "captions". A tiny image encoder and tiny text encoder trained jointly with the InfoNCE loss. The 4×4 similarity matrix should pull the diagonal (matched pairs) to high values and push the off-diagonal entries low.

The image-language interface — projector designs

Once the vision encoder is aligned, every patch becomes a ~1024-dim feature vector. A 224×224 image at patch 14 has 256 such tokens; the LLM might want them in 4096-dim hidden space and may not want all 256. The projector is the small network between the encoder and the LLM that does both jobs. Four designs dominate practice:

• Linear / 2-layer MLP (LLaVA-1.0, LLaVA-1.5). Token-for-token: each patch embedding maps to one image token in LLM space. Cheapest, surprisingly strong, but image-token count = patch count which dominates context length at high resolution.
• Perceiver resampler / Q-Former (Flamingo, BLIP-2). $K$ learned query tokens cross-attend to the patch features; output is exactly $K$ image tokens regardless of image resolution. Compute and context savings: ~32 query tokens instead of 576. Cost: a separately trained module.
• Cross-attention into the LLM (Flamingo, IDEFICS). Insert dedicated cross-attention layers inside the LLM that attend to image tokens. The LLM "sees" the image without spending self-attention budget on it.
• Token packing + AnyRes / dynamic tiling (LLaVA-NeXT, Qwen-VL-2.5, GPT-4o, Pixtral). High-resolution images are split into tiles; each tile is a sub-image; tile features are concatenated. Lets the model see a 1344×1344 image without a 1344-pixel-aware encoder.

The choice has measurable consequences:

Stage 2 — Image-grounded language pretraining

With the encoder aligned, you bolt on the language model. The projector is randomly initialised; the encoder and LLM are loaded pretrained.

Training data: image-caption pairs. Loss: standard next-token cross-entropy on the caption, with image tokens as a prefix context. Encoder and LLM are frozen; only the projector trains. This is cheap (60M params), runs on a handful of GPUs, and produces a model that can describe an image but can't follow much instruction.

Stage 3 — Instruction tuning

Now the model can describe images but can't answer questions about them or follow instructions. Stage 3 fixes that with a curated dataset of triples $(\text{image}, \text{instruction}, \text{response})$. Examples: "Count the people in this photo." "What's the temperature reading on this thermometer?" "Write Python that plots the chart in this image."

Loss: cross-entropy on the response tokens only (the image and instruction tokens are masked out). The projector continues to train; the LLM is now also trained, but with a low learning rate (LoRA on top of the LLM is the most common implementation).

Stage 4 — Preference alignment (RLHF / DPO)

The instruction-tuned model still says wrong things, hedges, or refuses unhelpfully. Stage 4 collects preference pairs (human, or from a stronger model) of "this answer is better than that one" and aligns the model to those preferences.

• RLHF. Train a reward model on preference pairs; fine-tune the VLM with PPO maximising the reward minus a KL penalty back to the SFT model.
• DPO (Direct Preference Optimization). A closed-form alternative that skips the reward model: optimise the implicit reward directly. Cheaper, often comparable.
• RLAIF. Use a stronger model as the preference labeller. Now the dominant recipe for mid-sized open VLMs.

Fine-tuning a VLM — recipes that actually work

Most labs don't train a VLM; they fine-tune one. The options form a ladder of cost vs flexibility:

• Prompt engineering only. Free; works for zero-shot tasks the base model already covers.
• Retrieval-augmented (RAG). Add a vector database of domain images / captions; retrieve and prepend at inference. No training; helps with tail knowledge.
• LoRA on the LLM (the default). Adapt $r$-rank low-rank deltas on the LLM's attention and MLP weights. ~1% of LLM parameters; fits on a single GPU. Common targets: $\{\,W_q, W_v\,\}$ at minimum; $\{\,W_q, W_k, W_v, W_o, W_{up}, W_{down}, W_{gate}\,\}$ for stronger adaptation.
• Full fine-tune of projector + LLM (LoRA-free). Best quality, multi-GPU. Needed when the domain shift is large (medical images, satellites, thermal).
• Full encoder + projector + LLM. Required when the modality is fundamentally new (depth, audio, point clouds). Cost: ~500-2000 GPU-hours on $\le$1B-param base.

Practical recipe (single-GPU lab fine-tune)

• Backbone. Llama-3.2-Vision-11B or Qwen-VL-7B as the frozen-encoder base. Both ship instruction-tuned weights.
• Adapter. LoRA rank $r = 16$ on $\{W_q, W_k, W_v, W_o\}$ in every layer; $\alpha = 32$; dropout $0.05$.
• Data. 5–50k domain image-instruction-answer triples. Synthesise instructions with a stronger VLM if you have only image-label pairs. Mix in 5–10% of the base instruction data to prevent catastrophic forgetting.
• Optimiser. AdamW, learning rate $1{\times}10^{-4}$ for LoRA params, $1{\times}10^{-5}$ for the projector (if unfrozen). Cosine schedule with 3% warmup, 1–3 epochs.
• Eval. Held-out val with the same prompt distribution; track exact-match, BLEU/Rouge for free-form, and a held-out hallucination probe.
• Bugs to watch. Frozen image tokens still consuming context budget; mismatched image preprocessing between train and eval (the silent killer); tokenizer template drift between base and your prompts.

Evaluation — what each benchmark actually measures

A trained VLM number means nothing without knowing the eval. The 2024-2026 standards:

Pick 2 broad benchmarks (MMBench + MMMU) and 1 hallucination probe (POPE) as the lab's house metrics; report all three on every checkpoint. The most common deception: "we beat GPT-4 on VQAv2" — VQAv2 has been saturated since 2022 and almost nothing publishes there anymore.

Image-text retrieval — recall@K, live

The CLIP-aligned encoders also give you free retrieval: take a text query, embed it, find the nearest image embeddings. Recall@K is the fraction of test queries whose true image appears in the top-K retrieved. Watch the live mini-CLIP from Step 1 produce its own retrieval matrix.

The thermal-image trap (and the tool-use fix)

Real VLMs fail spectacularly on thermal imagery. The reason is calibration: a thermal sensor outputs raw temperature per pixel, which is then mapped to a colour-map for human viewing. Two thermal images that look identical to a VLM (same false-colour palette) can correspond to wildly different absolute temperatures — the VLM only sees colours, not the metadata.

The fix isn't a bigger VLM. It's a tool-use loop:

1. VLM looks at the image.
2. VLM emits a tool call: "give me the temperature at pixel (i, j)".
3. Tool reads the raw radiometric TIFF and returns a number in Kelvin.
4. VLM reasons over numbers, not colours.

This is exactly the line of research behind tool-augmented thermal VLMs and the thermal-VLM benchmarking benchmark — concrete demonstration that the dominant VLMs (GPT-4, Claude, Gemini) fail on tasks where a tool-augmented smaller model succeeds. The lesson generalises: any modality whose pixels-look-the-same-but-mean-different-things (thermal, hyperspectral, depth, audio spectrograms) wants a tool layer.