Learning outcomes

By the end of this lecture you will be able to:

1. Turn an image into tokens (ViT patch embedding).
2. Explain CLIP contrastive training over image-text pairs.
3. Use CLIP for zero-shot classification.
4. Describe LLaVA's linear-projection bridge to an LLM.
5. Contrast native vs bolt-on multimodal architectures.
6. Diagnose VLM hallucination (language-prior override).

Where we are

• CNNs (L7–L9) · vision-specific inductive bias.
• Transformers (L13–L14) · originally for text.
• Self-supervised (L17) · contrastive and masked pretraining.

Today: what if one model handled both text AND vision?

Four questions:

1. How do Transformers process images (no convolutions)?
2. What did CLIP unlock?
3. How does LLaVA give an LLM eyes?
4. What's the 2026 multimodal state?

The 2020 bet

Controversial at the time — CNNs had reigned for 8 years. The bet: if you have enough data and compute, the right architecture is the one with the fewest inductive biases.

It worked. ViT-Huge pretrained on 300M images beat CNN SOTA on ImageNet by 2021.

Why throw away CNNs · the bet of 2020

The wager · with enough data, a model that has fewer prior assumptions but more capacity will outlearn a hand-engineered architecture. By 2021 (ViT-Huge on 300M images) the bet paid off · ViT matched ResNet on ImageNet.

How can a Transformer "read" an image?

From pixels to tokens · with a tiny example

A 4×4 grayscale image, 2×2 patches. We get patches.

Step 1 · patchify. Patch 1 (top-left) = .
Step 2 · flatten. Patch 1 → .
Step 3 · linear-project. Multiply by learned (size for output dim 3) → 3-d embedding.
Step 4 · prepend [CLS], add position embeddings. Sequence: [CLS, p1, p2, p3, p4] + [pe_0, pe_1, …].

For real 224×224 RGB with 16×16 patches:

• Each patch · numbers.
• Project to . Patches per image · .
• Final sequence · tokens. Standard Transformer from here.

The patchify step is conceptually a strided conv with kernel = stride = patch size.

Worked numeric · patch embedding

Tiny example. Flattened patch · .

Learned (size ):

Compute. .

• Component 1 ·
• Component 2 ·
• Component 3 ·

— the first "token" the Transformer sees, representing the top-left of the image.

From image to sequence · picture

How ViT works

ViT vs CNN · inductive biases

ViT variants you'll encounter

Swin Transformer (Liu et al. 2021) adds hierarchical windowing — bridges ViT and CNN. Popular in practice.

ViT in PyTorch · 30 lines

class ViT(nn.Module):
    def __init__(self, image_size=224, patch=16, dim=768, depth=12, heads=12, classes=1000):
        super().__init__()
        self.patch_embed = nn.Conv2d(3, dim, kernel_size=patch, stride=patch)   # patchify
        n_patches = (image_size // patch) ** 2
        self.cls_token = nn.Parameter(torch.randn(1, 1, dim))
        self.pos_embed = nn.Parameter(torch.randn(1, n_patches + 1, dim))
        self.blocks = nn.ModuleList([TransformerBlock(dim, heads) for _ in range(depth)])
        self.norm = nn.LayerNorm(dim)
        self.head = nn.Linear(dim, classes)

    def forward(self, x):
        x = self.patch_embed(x).flatten(2).transpose(1, 2)      # (B, N, D)
        cls = self.cls_token.expand(x.size(0), -1, -1)
        x = torch.cat([cls, x], dim=1) + self.pos_embed
        for b in self.blocks: x = b(x)
        return self.head(self.norm(x[:, 0]))                    # CLS → class logits

Literally a Transformer from L13 with a patch-embed preamble. No vision-specific ops.

CLIP · training as contrast matrix

CLIP · dual encoder

CLIP · the core idea

Train on (image, caption) pairs scraped from the web. Push matching pairs close in embedding space, push mismatched pairs far apart.

The training objective is InfoNCE (L17), extended across two modalities instead of two augmentations.

CLIP · the matching-game derivation

Tiny batch · image-text pairs .

1. Step 1 · embed. Encode each image and text → unit vectors and .
2. Step 2 · similarity matrix. .
   
   Diagonal = correct matches → maximize. Off-diagonal → minimize.
3. Step 3 · scale by temperature. .
4. Step 4 · row-wise CE. Each row of logits → softmax → CE against the diagonal label. (Image should match text .)
5. Step 5 · column-wise CE. Same for columns (text should match image ).
6. Final loss. .

• Image encoder · ViT-L/14 (or ResNet-50 for small).
• Text encoder · 12-layer Transformer.
• Data · 400M image-text pairs from the web. Batch size 32,768.

Worked numeric · CLIP loss for one row

. Normalized embeddings:

• , (good match)
• , (good match)
• .

Similarity matrix.

• ✓
• 
  

Logits row 1. .

Softmax. . Normalize → .

Loss row 1 (correct = index 0). — very low.

If the off-diagonal were higher, the loss spikes immediately. CLIP just runs this over 32k images simultaneously.

CLIP's killer feature · zero-shot

import clip
model, preprocess = clip.load("ViT-L/14")

image = preprocess(pil_image).unsqueeze(0)
texts = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
text_tokens = clip.tokenize(texts)

with torch.no_grad():
    img_emb  = model.encode_image(image)
    txt_embs = model.encode_text(text_tokens)

# Cosine similarity → pick the closest text
sims = (img_emb @ txt_embs.T).softmax(dim=-1)
# sims = [[0.89, 0.08, 0.03]]  → "cat"

No training on cats. CLIP has never seen an ImageNet label. Yet it beats ResNet-50 on zero-shot ImageNet classification.

Worked example · zero-shot a single image

Image · a photo of a tabby cat. CLIP image encoder produces a 768-dim vector (after L2-normalize · unit length).

Pick argmax · "cat" wins with 89% confidence. Total inference cost · 1 image-encoder call + 3 text-encoder calls. No fine-tuning. Add 100 classes · still 100 text-encoder calls (a few seconds, total) and you're done.

Why CLIP mattered

1. Universal image representation — one encoder works on any domain.
2. Prompt engineering for vision — "a photo of X" vs "a sketch of X" vs "a satellite image of X" shifts behavior without retraining.
3. Foundation for everything that came next — Stable Diffusion uses CLIP's text encoder; DALL-E 2, Flamingo, LLaVA all build on CLIP features.

Prompt engineering · for vision

Zero-shot CLIP is sensitive to how you phrase the class label. Changing the prompt template affects accuracy by ~10% absolute on ImageNet.

LLaVA · full stack

LLaVA · the architecture

image → ViT-L/14 (CLIP encoder) → 256 image tokens (frozen)
                                              ↓
                                     Linear projection
                                              ↓
                                       "Image tokens" in LLM space
                                              ↓
                          Llama 2 / Vicuna (autoregressive)
                                              ↓
                                       text response

Surprisingly good. The LLM brings reasoning; CLIP brings vision understanding; the projection layer glues them.

LLaVA · the translator analogy

LLaVA · the linear projection, with shapes

1. CLIP output · 256 patch embeddings, each 1024-d. Tensor [256, 1024].
2. LLM expects · token embeddings of dim 4096.
3. Bridge · linear layer W of shape [1024, 4096] plus bias [4096].

Per patch: llm_token = patch @ W + b. Shape check · [1, 1024] @ [1024, 4096] → [1, 4096] ✓.

Do this for all 256 patches → 256 vectors that "look like" tokens to the LLM. Prepend them to the user's text:

Stage 1 · freeze CLIP and LLM, train only , on caption data.
Stage 2 · unfreeze the LLM, fine-tune on instruction data.

New params · ~10M in the projection. A 7B LLM becomes multimodal with < 0.15% extra weights.

Worked numeric · the projection

CLIP patch · (3-d for clarity). LLM expects 4-d tokens.

Compute :

The LLM treats this just like the embedding for the word "cat".

Flamingo · cross-attention bridge

Alayrac et al. 2022 · an alternative approach:

• Keep the LLM frozen.
• Insert cross-attention layers between LLM blocks that attend to image features.
• Use a Perceiver Resampler — a learned set of query tokens that distill image features.

Benefit: the LLM never "sees" images as tokens; it queries them via cross-attention when it needs to.

Native-multimodal vs bolt-on · adopt or raise?

Native vs bolt-on · trade-offs

Cross-attention · Flamingo's design

Instead of converting the image into LLM-space tokens, Flamingo keeps vision features separate and injects them via cross-attention layers inserted between LLM blocks.

2026 multimodal state

Key trend: the input side is increasingly "anything", the output is still mostly text. True any-to-any (text→image→video→audio) is the 2026+ frontier.

Practical multimodal prompting

from anthropic import Anthropic
client = Anthropic()

response = client.messages.create(
    model="claude-4",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image", "source": {"type": "url", "url": "https://..."}},
                {"type": "text", "text": "What does this plot suggest?"}
            ]
        }
    ]
)

Three lines in; a paragraph of reasoning out. The API abstracts away all the CLIP + projection + LLM machinery.

VLM benchmarks · what we measure

Claude / GPT-4o / Gemini all push 85%+ on VQAv2 now. MMMU remains hard (50-60%) · genuine multi-modal reasoning is the frontier.

Hallucination · priors fighting evidence

This is a battle between prior (text statistics from training) and evidence (current visual input). When evidence is strong (clear photo), priors don't matter much. When evidence is weak, they take over and the model "fills in" things confidently.

Why VLMs hallucinate

Vision-language models sometimes describe things that aren't in the image:

Image · a cat on a mat.
Prompt · "describe the room in detail."
Output · "... next to a blue vase on the wooden table ..."

GPT-4o and Claude 3.5+ dramatically reduced hallucination via more curated training and RLHF specifically on vision tasks. Still not solved.

Emerging applications

The agentic side (Claude computer use, GPT operator) is where multimodal is most valuable in 2026.