FROM CLIP TO
VISION-LANGUAGE MODELS

A global vector can be useful, but fine-grained tasks motivate exposing richer visual states. This is a design motivation, not an impossibility theorem.

Ask: Would one global CLIP vector always be enough?

Answer: A global vector can be useful, but fine-grained tasks motivate exposing richer visual states. This is a design motivation, not an impossibility theorem.

Preload the model before class. First download needs network; recorded outputs are available offline. Run the dog, spatial and absent-object cases.

Ask: A reference answer is not a model prediction.

Answer: Preload the model before class. First download needs network; recorded outputs are available offline. Run the dog, spatial and absent-object cases.

Vectors are normalized for the cosine-style comparison. CLIP is a VLM broadly; here we distinguish contrastive matching from generative VLMs.

Ask: Image vector u · text vector v · similarity uᵀv

Answer: Vectors are normalized for the cosine-style comparison. CLIP is a VLM broadly; here we distinguish contrastive matching from generative VLMs.

The image encoder remains useful. But a matching score alone cannot produce a next-token vocabulary distribution.

Ask: Desired output: ? ? ?

Answer: The image encoder remains useful. But a matching score alone cannot produce a next-token vocabulary distribution.

Recall the decoder-only model from Beyond Attention before attaching vision.

Ask: A causal language model maps a prefix to a next-token distribution.

Answer: Recall the decoder-only model from Beyond Attention before attaching vision.

Keep the two branches disconnected. Do not draw a connector until we derive what it must do.

Ask: How can visual states enter the language computation?

Answer: Keep the two branches disconnected. Do not draw a connector until we derive what it must do.

Spend roughly five minutes on the three preceding idea slides, with no matrix shapes or attention arithmetic. Ask students to explain each idea in one sentence. Now return to Idea 1 and derive it. Ideas 2 and 3 get their own later deep dives.

Ask: Must a model choose only one of these three ideas?

Answer: No. Prefix and separate memory are two ways to access vision. Compression can be combined with either; for example, a short visual summary can be prepended to the text.

Trace all N patch rows through the amber projector into the input row. Trace question and answer-so-far tokenization and embedding into the same row. The answer is initially empty, then grows by generated tokens. V1, V2, …, VN abbreviates the full visual sequence. Point to causal self-attention, MLP and vocabulary head from the previous lecture. Positions, normalization and exact chat markers are omitted. We will now unpack only the new visual path.

Ask: One sequence; the same causal self-attention can use both kinds of context.

Answer: Trace all N patch rows through the amber projector into the input row. Trace question and answer-so-far tokenization and embedding into the same row. The answer is initially empty, then grows by generated tokens. V1, V2, …, VN abbreviates the full visual sequence. Point to causal self-attention, MLP and vocabulary head from the previous lecture. Positions, normalization and exact chat markers are omitted. We will now unpack only the new visual path.

Return to this three-job plan after the projector calculation and after the mask. Students should be able to point to each job in the preceding complete architecture.

Ask: We already have the decoder. Now build its visual input.

Answer: Return to this three-job plan after the projector calculation and after the mask. Students should be able to point to each job in the preceding complete architecture.

The numbered 3 × 4 grid is an illustrative twelve-patch example, not a model configuration. All twelve patch rows are drawn, in row-major order matching the image labels. Each box denotes one whole feature vector, not one scalar coordinate. ViT embeds the patches, adds positions, and updates their states with full self-attention. The row at patch position i can therefore contain information from the whole image. The CLS row is introduced on the next slide.

Ask: Patch 1 has row h1, patch 2 has row h2, and so on through patch 12.

Answer: The numbered 3 × 4 grid is an illustrative twelve-patch example, not a model configuration. All twelve patch rows are drawn, in row-major order matching the image labels. Each box denotes one whole feature vector, not one scalar coordinate. ViT embeds the patches, adds positions, and updates their states with full self-attention. The row at patch position i can therefore contain information from the whole image. The CLS row is introduced on the next slide.

Use the same twelve numbered image patches and twelve output rows as the previous slide. This standard ViT also has a CLS token, introduced before the encoder; its output state provides a whole-image summary. CLS is an extra sequence position, not a thirteenth image patch or a separate encoder. The diagram abstracts patch embedding, positions, and the input CLS token inside the ViT path. Its twelve patch states are contextual vectors, not isolated patch descriptors. For the visual-prefix design being developed, retain all twelve patch rows and then project each one to the decoder width; the CLS state is not included in that patch sequence. Twelve is the complete illustrative example, not a universal model count. Keep this explanation about ViT outputs; the earlier CLIP matching discussion need not be repeated.

Ask: Here: 1 CLS row + 12 patch rows. We keep all 12 patch rows.

Answer: Use the same twelve numbered image patches and twelve output rows as the previous slide. This standard ViT also has a CLS token, introduced before the encoder; its output state provides a whole-image summary. CLS is an extra sequence position, not a thirteenth image patch or a separate encoder. The diagram abstracts patch embedding, positions, and the input CLS token inside the ViT path. Its twelve patch states are contextual vectors, not isolated patch descriptors. For the visual-prefix design being developed, retain all twelve patch rows and then project each one to the decoder width; the CLS state is not included in that patch sequence. Twelve is the complete illustrative example, not a universal model count. Keep this explanation about ViT outputs; the earlier CLIP matching discussion need not be repeated.

Return to the Newfoundland image on the next frame. This invoice explains the design motivation. Keeping patch states does not guarantee OCR success; resolution and training still matter.

Ask: Give language access to more visual evidence before asking it to answer.

Answer: Return to the Newfoundland image on the next frame. This invoice explains the design motivation. Keeping patch states does not guarantee OCR success; resolution and training still matter.

Use “row” and “vector” interchangeably here. Attention has mixed image context into each row. Do not label one row “dog” or claim that one coordinate is fur colour.

Ask: Rows are vectors, not object labels or named semantic attributes.

Answer: Use “row” and “vector” interchangeably here. Attention has mixed image context into each row. Do not label one row “dog” or claim that one coordinate is fur colour.

This is the connection to the earlier text architecture: tokenization selects embedding rows. Image vectors bypass a word-ID lookup and enter at the vector interface. They also need the implementation’s positional treatment.

Ask: Text IDs become embeddings before reaching the Transformer blocks.

Answer: This is the connection to the earlier text architecture: tokenization selects embedding rows. Image vectors bypass a word-ID lookup and enter at the vector interface. They also need the implementation’s positional treatment.

These are illustrative model widths. The operation acts on the last dimension, not the row dimension.

Ask: 576 rows × 1,024 coordinates become how many rows × 4,096 coordinates?

Answer: 576 rows. The projector changes each vector’s width; it does not compress or select patch positions.

If a student answered 32, explain that they have proposed an extra summarization operation. The simple LLaVA-style row-wise projection does not do that.

Ask: We will learn how to reduce the row count in the compression section.

Answer: If a student answered 32, explain that they have proposed an extra summarization operation. The simple LLaVA-style row-wise projection does not do that.

Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Ask: Visual vectors and text embeddings share the decoder’s input width.

Answer: Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Ask: Visual vectors and text embeddings share the decoder’s input width.

Answer: Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Ask: Visual vectors and text embeddings share the decoder’s input width.

Answer: Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Ask: Visual vectors and text embeddings share the decoder’s input width.

Answer: Simplified chat template. Actual image markers, role markers and positional treatment depend on the implementation.

Read one query row first. The current position may also attend to itself; the simplified row here emphasizes earlier context.

Ask: The assistant’s state can use both the picture and the question.

Answer: Read one query row first. The current position may also attend to itself; the simplified row here emphasizes earlier context.

V1–V3 are the projected image vectors called z1…zN in the earlier numeric example; three visual positions make the mask readable. Rows are queries and columns are keys. Read the first three rows before the text: V1 can read V1, V2 can read V1 and V2, and V3 can read V1 through V3. No visual decoder query reads later visual positions or later text, but every visual input already incorporates image-wide context from the ViT encoder. Each text row can read all preceding visual rows, preceding text rows and itself. All supplied image/prompt rows are processed together with this mask during prefill, not generated one by one. New answer tokens are generated autoregressively afterwards. This is the ordinary LLaVA/LLaMA causal-mask example; some architectures instead use full attention within the visual prefix. The mask controls attention edges inside the decoder, not the ViT encoder.

Ask: Colored: allowed · grey: blocked. The triangle includes the visual positions.

Answer: V1–V3 are the projected image vectors called z1…zN in the earlier numeric example; three visual positions make the mask readable. Rows are queries and columns are keys. Read the first three rows before the text: V1 can read V1, V2 can read V1 and V2, and V3 can read V1 through V3. No visual decoder query reads later visual positions or later text, but every visual input already incorporates image-wide context from the ViT encoder. Each text row can read all preceding visual rows, preceding text rows and itself. All supplied image/prompt rows are processed together with this mask during prefill, not generated one by one. New answer tokens are generated autoregressively afterwards. This is the ordinary LLaVA/LLaMA causal-mask example; some architectures instead use full attention within the visual prefix. The mask controls attention edges inside the decoder, not the ViT encoder.

No extra cross-attention layer is required by this access pattern.

Ask: We have built all three jobs: visual rows, matching width, and a prefix.

Answer: No extra cross-attention layer is required by this access pattern.

A simple row-wise linear or MLP projector keeps the patch-row count. The few boxes are a drawn subset, not token selection. LLaVA variants differ in details. Later compression methods deliberately reduce the visual sequence.

Ask: Projected patch vectors occupy earlier positions; they do not become word IDs.

Answer: A simple row-wise linear or MLP projector keeps the patch-row count. The few boxes are a drawn subset, not token selection. LLaVA variants differ in details. Later compression methods deliberately reduce the visual sequence.

Trace the two inputs separately. The purple path embeds the question and tokens generated so far; the answer portion is empty initially. The vision encoder exposes h1, h2, …, hN, a sequence of feature rows. B × N × dV means batch size × retained visual rows per example × features per row; the drawing shows one image. Text states supply Q, image states are projected to K and V. The attention result updates the language states before the vocabulary head. N varies with image resolution, patch size, CLS handling and any compression; 197 is only a conditional example. This schematic shows the standard sequence-to-sequence mechanism, not Flamingo’s exact layer ordering or gates.

Ask: Causal self-attention reads text; cross-attention reads image memory.

Answer: Trace the two inputs separately. The purple path embeds the question and tokens generated so far; the answer portion is empty initially. The vision encoder exposes h1, h2, …, hN, a sequence of feature rows. B × N × dV means batch size × retained visual rows per example × features per row; the drawing shows one image. Text states supply Q, image states are projected to K and V. The attention result updates the language states before the vocabulary head. N varies with image resolution, patch size, CLS handling and any compression; 197 is only a conditional example. This schematic shows the standard sequence-to-sequence mechanism, not Flamingo’s exact layer ordering or gates.

Use the complete source-left, decoder-right architecture from the final diagrams of the earlier lecture. The current target state first uses causal target self-attention and then reads source states. Replace only the source modality on the next slide.

Ask: English source states stayed available while the Hindi prefix grew.

Answer: Use the complete source-left, decoder-right architecture from the final diagrams of the earlier lecture. The current target state first uses causal target self-attention and then reads source states. Replace only the source modality on the next slide.

The text sequence remains separate from visual memory. We have not yet computed a visual message.

Ask: The source changes. The decoder still reads a separate memory.

Answer: The text sequence remains separate from visual memory. We have not yet computed a visual message.

These phrases explain data flow, not interpretable concepts encoded by every learned coordinate. A query is a vector derived from the current text computation, not the entire question string. We now derive these three pieces on a zoomed-in view.

Ask: If we swap the image but keep this incoming language state fixed, what changes?

Answer: The visual keys and values change. The incoming query is fixed for this isolated lookup; in the full model later language states can also change.

Scaling controls dot-product magnitude before normalization. One attention head shown.

Ask: The result is one score per visual state.

Answer: Scaling controls dot-product magnitude before normalization. One attention head shown.

These are attention weights, not probabilities of answer words.

Ask: Softmax normalizes across the visual memory positions.

Answer: These are attention weights, not probabilities of answer words.

Heads are concatenated and projected in multi-head attention; the result returns to the language computation.

Ask: One query produces one context vector per head.

Answer: Heads are concatenated and projected in multi-head attention; the result returns to the language computation.

Return to the visual-memory architecture. We are inside its teal cross-attention box, computing one message for one language position. The three row labels and all numbers are hand-set, not measured regions from the dog image. A large attention weight does not by itself prove grounding.

Ask: Toy numbers and labels; no measured attention map is being shown.

Answer: Return to the visual-memory architecture. We are inside its teal cross-attention box, computing one message for one language position. The three row labels and all numbers are hand-set, not measured regions from the dog image. A large attention weight does not by itself prove grounding.

The numerical vector is not a word ID. Ask students to point to the vocabulary head in the whole architecture.

Ask: Does the cross-attention output directly name the answer token?

Answer: No. It updates a language hidden state. The vocabulary head later scores output token IDs.

Trace memory → cross-attention → updated decoder computation → vocabulary head. Cross-attention returns features, not an English answer. Model-specific blocks can repeat this lookup in depth.

Ask: Visual lookup changes the language state; the language head chooses words.

Answer: Trace memory → cross-attention → updated decoder computation → vocabulary head. Cross-attention returns features, not an English answer. Model-specific blocks can repeat this lookup in depth.

Same image and prompt. Neither access pattern is universally better; training, model size, compute and task matter.

Ask: In-sequence visual context or a separate visual memory.

Answer: Same image and prompt. Neither access pattern is universally better; training, model size, compute and task matter.

Compare with the first complete prefix architecture. Only the left-hand connector changes. The decoder still consumes visual vectors followed by text. The same compression idea can instead provide the separate memory used by cross-attention.

Ask: Replace the row-wise projector with a learned summary followed by projection.

Answer: Compare with the first complete prefix architecture. Only the left-hand connector changes. The decoder still consumes visual vectors followed by text. The same compression idea can instead provide the separate memory used by cross-attention.

Illustrative 24 by 24 patch grid, with no CLS state included. Token counts depend on image resolution and preprocessing.

Ask: 576 visual states + 64 text positions = 640 positions.

Answer: Illustrative 24 by 24 patch grid, with no CLS state included. Token counts depend on image resolution and preprocessing.

Both bars use the same scale. Dense attention score counts are 640² versus 96², about 44.4 times fewer pairs, not 44.4 times total speedup.

Ask: 32 visual states + 64 text positions = 96 positions.

Answer: Both bars use the same scale. Dense attention score counts are 640² versus 96², about 44.4 times fewer pairs, not 44.4 times total speedup.

Recall the lookup we just computed: one query returns one mixture of values. Here the initial query vectors are learned parameters instead of being derived from the current answer position. 32 query slots therefore produce 32 output vectors. The initial slots persist across images; the returned summaries change with the image. Real resamplers may refine these slots across layers. The picture shows 4 of 32 outputs and 6 of 576 inputs.

Ask: How can we read a fixed-size summary of a variable visual memory?

Answer: Recall the lookup we just computed: one query returns one mixture of values. Here the initial query vectors are learned parameters instead of being derived from the current answer position. 32 query slots therefore produce 32 output vectors. The initial slots persist across images; the returned summaries change with the image. Real resamplers may refine these slots across layers. The picture shows 4 of 32 outputs and 6 of 576 inputs.

Recall the lookup we just computed: one query returns one mixture of values. Here the initial query vectors are learned parameters instead of being derived from the current answer position. 32 query slots therefore produce 32 output vectors. The initial slots persist across images; the returned summaries change with the image. Real resamplers may refine these slots across layers. The picture shows 4 of 32 outputs and 6 of 576 inputs.

Ask: How can we read a fixed-size summary of a variable visual memory?

Answer: Recall the lookup we just computed: one query returns one mixture of values. Here the initial query vectors are learned parameters instead of being derived from the current answer position. 32 query slots therefore produce 32 output vectors. The initial slots persist across images; the returned summaries change with the image. Real resamplers may refine these slots across layers. The picture shows 4 of 32 outputs and 6 of 576 inputs.

Recall the lookup we just computed: one query returns one mixture of values. Here the initial query vectors are learned parameters instead of being derived from the current answer position. 32 query slots therefore produce 32 output vectors. The initial slots persist across images; the returned summaries change with the image. Real resamplers may refine these slots across layers. The picture shows 4 of 32 outputs and 6 of 576 inputs.

Ask: The query count fixes the number of output vectors.

Answer: Recall the lookup we just computed: one query returns one mixture of values. Here the initial query vectors are learned parameters instead of being derived from the current answer position. 32 query slots therefore produce 32 output vectors. The initial slots persist across images; the returned summaries change with the image. Real resamplers may refine these slots across layers. The picture shows 4 of 32 outputs and 6 of 576 inputs.

A resampler can feed either access pattern. Do not equate fewer visual tokens with universally better performance.

Ask: Compression is a separate choice from prefix versus cross-attention access.

Answer: A resampler can feed either access pattern. Do not equate fewer visual tokens with universally better performance.

This is a design comparison, not a ranking. BLIP-2 supports different language backbones; decoder-only examples use visual prefix conditioning. Training objectives and exact modules differ.

Ask: Flamingo and BLIP-2 use learned-query mechanisms in different systems.

Answer: This is a design comparison, not a ranking. BLIP-2 supports different language backbones; decoder-only examples use visual prefix conditioning. Training objectives and exact modules differ.

A pretrained but frozen weight is still a learned parameter. The frozen/trainable decision will be a separate overlay during training. Learned query embeddings are parameters; actual projected attention queries are activations.

Ask: Parameters persist across examples; activations depend on the input.

Answer: A pretrained but frozen weight is still a learned parameter. The frozen/trainable decision will be a separate overlay during training. Learned query embeddings are parameters; actual projected attention queries are activations.

Keep the canonical layout fixed and fade already-introduced stages. The photo and question are unchanged.

Ask: The encoder computes the visual state matrix.

Answer: Keep the canonical layout fixed and fade already-introduced stages. The photo and question are unchanged.

Keep the canonical layout fixed and fade already-introduced stages. The photo and question are unchanged.

Ask: The visual context is available to the language computation.

Answer: Keep the canonical layout fixed and fade already-introduced stages. The photo and question are unchanged.

Keep the canonical layout fixed and fade already-introduced stages. The photo and question are unchanged.

Ask: The current language state will drive the vocabulary head.

Answer: Keep the canonical layout fixed and fade already-introduced stages. The photo and question are unchanged.

Scores are symbolic; the browser API does not expose reliable per-token logits here. No numerical probabilities are invented.

Ask: Visual evidence can change hₜ and therefore every vocabulary score.

Answer: Scores are symbolic; the browser API does not expose reliable per-token logits here. No numerical probabilities are invented.

“dog” is an illustrative selection explaining the computation, not a claim that the tiny model chose it. The measured counterfactual follows. Real tokenizers may split displayed words.

Ask: Selection can be greedy or sampled; the output is a vocabulary token.

Answer: “dog” is an illustrative selection explaining the computation, not a claim that the tiny model chose it. The measured counterfactual follows. Real tokenizers may split displayed words.

Blank means a white image passed through the same image processor, not omission of the image input. Responses test sensitivity to image changes; they do not by themselves establish reliable grounding. Full exact records and metadata are in results/vlm_lab_outputs.json.

Ask: Actual saved answers from the pinned browser model; probabilities are not exposed.

Answer: Blank means a white image passed through the same image processor, not omission of the image input. Responses test sensitivity to image changes; they do not by themselves establish reliable grounding. Full exact records and metadata are in results/vlm_lab_outputs.json.

Shape matching made the forward computation possible. It did not teach what the image features mean to language. Use image/answer pairs, next-token loss and gradients. The next slides identify which parameters receive updates in example training recipes.

Ask: Use the correct answer tokens to train the visual-to-language connection.

Answer: Shape matching made the forward computation possible. It did not teach what the image features mean to language. Use image/answer pairs, next-token loss and gradients. The next slides identify which parameters receive updates in example training recipes.

The snowflake denotes frozen parameters; the flame in later slides denotes parameters being optimized.

Ask: The connector is new; vision and language already have learned weights.

Answer: The snowflake denotes frozen parameters; the flame in later slides denotes parameters being optimized.

Simplified LLaVA-style recipe. Frozen vision encoder and frozen LLM; connector trainable. Exact objectives and freeze choices are architecture-dependent.

Ask: Example recipe: caption alignment; freezing choices depend on the model.

Answer: Simplified LLaVA-style recipe. Frozen vision encoder and frozen LLM; connector trainable. Exact objectives and freeze choices are architecture-dependent.

Example recipe: frozen vision, trainable connector and LLM, or LLM adapters. Some models also unfreeze vision. These choices are model-dependent.

Ask: Example recipe: image + question + answer; train the LLM or its adapters.

Answer: Example recipe: frozen vision, trainable connector and LLM, or LLM adapters. Some models also unfreeze vision. These choices are model-dependent.

A denotes selected assistant answer targets. This answer is an authored training example. Dataset coverage matters: captions alone may not teach counting, OCR or justified rejection.

Ask: The loss supervises the answer targets selected by the training recipe.

Answer: A denotes selected assistant answer targets. This answer is an authored training example. Dataset coverage matters: captions alone may not teach counting, OCR or justified rejection.

Every displayed token has a target/no-target marker. The loss uses logits at the preceding positions to predict these targets; a prompt-position logit can predict the first answer. Masked context still influences answer gradients.

Ask: Only Black, period and EOS are targets; preceding-position logits predict them.

Answer: Every displayed token has a target/no-target marker. The loss uses logits at the preceding positions to predict these targets; a prompt-position logit can predict the first answer. Masked context still influences answer gradients.

The attention mask controls information flow; the loss mask selects supervised target positions. They are not interchangeable.

Ask: Removing a target loss does not hide a context token.

Answer: The attention mask controls information flow; the loss mask selects supervised target positions. They are not interchangeable.

Differentiate the answer loss through the frozen language operations with respect to their input, so connector weights receive a gradient. The LLM weights are not updated. Wrapping that LLM forward in no_grad would sever the required path.

Ask: Frozen ≠ stop-gradient.

Answer: Differentiate the answer loss through the frozen language operations with respect to their input, so connector weights receive a gradient. The LLM weights are not updated. Wrapping that LLM forward in no_grad would sever the required path.