From CLIP to Vision-Language Models

A global vector can be useful, but fine-grained tasks motivate exposing richer visual states. This is a design motivation, not an impossibility theorem.

Ask: Would one global CLIP vector always be enough?

Answer: A global vector can be useful, but fine-grained tasks motivate exposing richer visual states. This is a design motivation, not an impossibility theorem.

Preload the model before class. First download needs network; recorded outputs are available offline. Run the dog, spatial and absent-object cases.

Ask: A reference answer is not a model prediction.

Answer: Preload the model before class. First download needs network; recorded outputs are available offline. Run the dog, spatial and absent-object cases.

Reuse the decoder-only portrait and the Raghav goes to → school example from Beyond Attention. First reveal the known text. Then look up the token embeddings and supply position information. The drawing uses additive positions as the familiar teaching example; other decoders may implement positions with RoPE inside attention. Repeat L blocks of causal self-attention plus residual and MLP plus residual. The triangular permission matrix means each position can use itself and earlier positions. Every position gets an updated state, but to extend this prefix read only the last one, e3. Final normalization, where required by the model, is omitted from this recap. The vocabulary head maps the last state to scores for all vocabulary entries; a selection rule chooses the next token. The illustrative continuation is school. Append its token, look up its embedding and repeat with the longer prefix; a KV cache can reuse preceding computation. These are illustrative word-sized tokens, not measured tokenizer boundaries or a checkpoint output. No separate text encoder or cross-attention is needed for this text-only decoder. The source architecture is transformer-outputs/figures/decoder-use.svg, generated by decoder_portrait in its src/refinement_diagrams.py.

Ask: Read the last updated state → score the vocabulary → append a token → repeat.

Answer: Reuse the decoder-only portrait and the Raghav goes to → school example from Beyond Attention. First reveal the known text. Then look up the token embeddings and supply position information. The drawing uses additive positions as the familiar teaching example; other decoders may implement positions with RoPE inside attention. Repeat L blocks of causal self-attention plus residual and MLP plus residual. The triangular permission matrix means each position can use itself and earlier positions. Every position gets an updated state, but to extend this prefix read only the last one, e3. Final normalization, where required by the model, is omitted from this recap. The vocabulary head maps the last state to scores for all vocabulary entries; a selection rule chooses the next token. The illustrative continuation is school. Append its token, look up its embedding and repeat with the longer prefix; a KV cache can reuse preceding computation. These are illustrative word-sized tokens, not measured tokenizer boundaries or a checkpoint output. No separate text encoder or cross-attention is needed for this text-only decoder. The source architecture is transformer-outputs/figures/decoder-use.svg, generated by decoder_portrait in its src/refinement_diagrams.py.

Carry the same causal attention / MLP / last-state / vocabulary-head grammar forward from Beyond Attention. Now the text prompt is Describe this dog. The ViT image branch uses full self-attention and exposes N contextual feature rows; h1, h2, …, hN denotes all retained rows, with middle rows abbreviated. The text branch tokenizes the prompt and looks up embeddings with position information; there is no separate text encoder. For clarity this drawing shows only the last updated output state, although the blocks update every position. Word-sized tokens are illustrative. Normalization, tokenizer IDs and exact position implementation are omitted here. Keep the two branches disconnected: the decoder can already generate tokens but has not yet been given the image features. Do not imply that an unconnected language decoder can see the dog. The three following idea diagrams show the alternative connections before we derive their details.

Ask: We have both components. Next, choose how the decoder accesses the visual rows.

Answer: Carry the same causal attention / MLP / last-state / vocabulary-head grammar forward from Beyond Attention. Now the text prompt is Describe this dog. The ViT image branch uses full self-attention and exposes N contextual feature rows; h1, h2, …, hN denotes all retained rows, with middle rows abbreviated. The text branch tokenizes the prompt and looks up embeddings with position information; there is no separate text encoder. For clarity this drawing shows only the last updated output state, although the blocks update every position. Word-sized tokens are illustrative. Normalization, tokenizer IDs and exact position implementation are omitted here. Keep the two branches disconnected: the decoder can already generate tokens but has not yet been given the image features. Do not imply that an unconnected language decoder can see the dog. The three following idea diagrams show the alternative connections before we derive their details.

Spend roughly five minutes on the three preceding idea slides, with no matrix shapes or attention arithmetic. Ask students to explain each idea in one sentence. Now return to Idea 1 and derive it. Ideas 2 and 3 get their own later deep dives.

Ask: Must a model choose only one of these three ideas?

Answer: No. Prefix and separate memory are two ways to access vision. Compression can be combined with either; for example, a short visual summary can be prepended to the text.

Trace all N patch rows through the amber projector into the input row. Trace question and answer-so-far tokenization and embedding into the same row. The answer is initially empty, then grows by generated tokens. V1, V2, …, VN abbreviates the full visual sequence. Point to causal self-attention, MLP and vocabulary head from the previous lecture. Positions, normalization and exact chat markers are omitted. We will now unpack only the new visual path.

Ask: One sequence; the same causal self-attention can use both kinds of context.

Answer: Trace all N patch rows through the amber projector into the input row. Trace question and answer-so-far tokenization and embedding into the same row. The answer is initially empty, then grows by generated tokens. V1, V2, …, VN abbreviates the full visual sequence. Point to causal self-attention, MLP and vocabulary head from the previous lecture. Positions, normalization and exact chat markers are omitted. We will now unpack only the new visual path.

Return to this three-job plan after the projector calculation and after the mask. Students should be able to point to each job in the preceding complete architecture.

Ask: We already have the decoder. Now build its visual input.

Answer: Return to this three-job plan after the projector calculation and after the mask. Students should be able to point to each job in the preceding complete architecture.

The numbered 3 × 4 grid is an illustrative twelve-patch example, not a model configuration. All twelve patch rows are drawn, in row-major order matching the image labels. Each box denotes one whole feature vector, not one scalar coordinate. ViT embeds the patches, adds positions, and updates their states with full self-attention. The row at patch position i can therefore contain information from the whole image. The CLS row is introduced on the next slide.

Ask: Patch 1 has row h1, patch 2 has row h2, and so on through patch 12.

Answer: The numbered 3 × 4 grid is an illustrative twelve-patch example, not a model configuration. All twelve patch rows are drawn, in row-major order matching the image labels. Each box denotes one whole feature vector, not one scalar coordinate. ViT embeds the patches, adds positions, and updates their states with full self-attention. The row at patch position i can therefore contain information from the whole image. The CLS row is introduced on the next slide.

Use the same twelve numbered image patches and twelve output rows as the previous slide. This standard ViT also has a CLS token, introduced before the encoder; its output state provides a whole-image summary. CLS is an extra sequence position, not a thirteenth image patch or a separate encoder. The diagram abstracts patch embedding, positions, and the input CLS token inside the ViT path. Its twelve patch states are contextual vectors, not isolated patch descriptors. For the visual-prefix design being developed, retain all twelve patch rows and then project each one to the decoder width; the CLS state is not included in that patch sequence. Twelve is the complete illustrative example, not a universal model count. Keep this explanation about ViT outputs; the earlier CLIP matching discussion need not be repeated.

Ask: Forward all 12 patch rows to the projector; leave out the CLS output.

Answer: Use the same twelve numbered image patches and twelve output rows as the previous slide. This standard ViT also has a CLS token, introduced before the encoder; its output state provides a whole-image summary. CLS is an extra sequence position, not a thirteenth image patch or a separate encoder. The diagram abstracts patch embedding, positions, and the input CLS token inside the ViT path. Its twelve patch states are contextual vectors, not isolated patch descriptors. For the visual-prefix design being developed, retain all twelve patch rows and then project each one to the decoder width; the CLS state is not included in that patch sequence. Twelve is the complete illustrative example, not a universal model count. Keep this explanation about ViT outputs; the earlier CLIP matching discussion need not be repeated.

Answer the selection question explicitly: use the patch rows alone, not CLS plus patch rows. The CLS token is still present inside the ViT; only its output row is excluded at the connector interface. The twelve displayed patch rows are the complete illustrative set, not a count measured from a model processing this invoice. In general N follows from the retained image patch grid. The invoice asks for an exact ID and a numerical relationship, motivating access to multiple spatially indexed feature vectors rather than routing the entire image through only one global summary. All patch rows are contextual after full self-attention; they are not isolated pixel crops or object labels. Keeping them gives the decoder access to these separate features but does not guarantee OCR or fine-detail accuracy. This is the LLaVA-style patch-only connection being taught, not a rule that every VLM must discard CLS. Original LLaVA feature_select in patch mode keeps image_features[:, 1:], while cls_patch is an alternative. Return to the Newfoundland image on the next frame.

Ask: Do we forward CLS plus the patch rows, or just the patch rows?

Answer: Just the N patch output rows in this design. CLS participates inside the vision encoder, then its output row is omitted. Every retained patch row goes through the projector and becomes one visual-prefix vector. Other architectures may make a different selection.

Revisit the same dog and complete twelve-patch grid as the preceding ViT recap. Read the image in row-major order: all twelve patches become twelve input tokens after patch embedding and position information. The full-attention ViT updates every token using image context. Stack all twelve retained output patch rows into Hᵥ. Every matrix cell hᵢ,ⱼ is one scalar feature; hᵢ is the complete row vector, not one scalar or an object label. N counts rows (visual token positions); dᵥ counts columns (features per token). Highlight patch 5, input token p5, and output row h5 to show position correspondence, while emphasizing that full attention makes h5 contextual rather than an isolated patch descriptor. This illustration uses N=12 and dᵥ=2 to show every row and column, consistent with the two-coordinate worked example that follows. These are illustrative dimensions, not actual model widths or measured features. No patch rows are omitted. CLS is excluded from the retained patch matrix and the batch axis is omitted because we show one image. With a batch, the shape is B × N × dᵥ.

Ask: In the drawn example, how many visual tokens and how many features does each output token have?

Answer: Twelve retained visual tokens give N=12 rows. Each output row has dᵥ=2 feature values, so Hᵥ is 12 × 2. h5 is the whole fifth row, whose two cells are h5,1 and h5,2. Attention allows that row to contain information from other image patches.

This is the connection to the earlier text architecture: tokenization selects embedding rows. Image vectors bypass a word-ID lookup and enter at the vector interface. They also need the implementation’s positional treatment.

Ask: Text IDs become embeddings before reaching the Transformer blocks.

Answer: This is the connection to the earlier text architecture: tokenization selects embedding rows. Image vectors bypass a word-ID lookup and enter at the vector interface. They also need the implementation’s positional treatment.

Stay with the example already on the board. Twelve is derived from the complete 3 × 4 patch grid shown again here. Two is the illustrative vision feature width in Hᵥ; three is the illustrative text embedding and decoder-input width. The widths are teaching choices from the preceding worked example, not quantities measured from the dog image. Wₚ has two rows and three columns so it maps every two-feature image row to a three-feature vector. Do not introduce realistic model-scale widths at this checkpoint.

Ask: Our image has 3 × 4 patches. Hᵥ is 12 × 2 and Wₚ is 2 × 3. What is the shape of Z?

Answer: 12 × 3. Twelve comes from the image patch count. Two and three are the illustrative input and output feature widths from our worked example. Row-wise projection retains all twelve visual token positions.

The source panels and matrix shapes stay in the same places as on the question slide. Read (12 × 2)(2 × 3) = 12 × 3. The shared inner dimension two makes the multiplication valid; twelve rows remain, and each output row has three coordinates. A lower row count would require an additional compression operation, introduced later. These are the same illustrative feature widths, not a newly introduced model configuration.

Ask: Which dimensions change, and where did they come from?

Answer: N=12 comes from the 3 × 4 image grid and stays fixed. The feature width changes from the chosen vision width dᵥ=2 to the chosen decoder/text width dₗ=3.

Continue the same 3 × 4 image and 2-to-3 feature-width example. All twelve projected vectors are drawn. They are inserted as vectors rather than looked up from word vocabulary IDs. The next slides explain the formatted text part.

Ask: Keep all 12 patch rows; each row has the decoder’s input width.

Answer: Continue the same 3 × 4 image and 2-to-3 feature-width example. All twelve projected vectors are drawn. They are inserted as vectors rather than looked up from word vocabulary IDs. The next slides explain the formatted text part.

This is a teaching chat format, not an exact LLaVA serialization. USER and ASST are readable stand-ins. Some models have dedicated single vocabulary tokens for roles; others spell roles using ordinary tokens and separators, so one role label need not equal one token. Image placement, delimiters and tokenization depend on the trained chat template. Keep the same template at training and inference. The user marker begins the human message; the assistant marker cues the model response. Both are already in the inference prompt before generating the first answer token. It is an illustrative first token, not a measured prediction. Chat training teaches the model how to respond to these structural cues.

Ask: Does the model have to generate ASST before it can answer?

Answer: In this illustrated template, no. The application adds the assistant-start marker to the prompt. The model predicts the first answer token from the final supplied position.

This is a teaching chat format, not an exact LLaVA serialization. USER and ASST are readable stand-ins. Some models have dedicated single vocabulary tokens for roles; others spell roles using ordinary tokens and separators, so one role label need not equal one token. Image placement, delimiters and tokenization depend on the trained chat template. Keep the same template at training and inference. For this single-token illustration, idᵤ and idₐ are symbolic vocabulary indices, not extra numerical features. E has vocabulary-size rows and dL columns; only three selected rows are shown. The blank cells stand for learned coordinates. Existing tokens reuse checkpoint embeddings. When training from scratch, embedding weights are initialized once; adding tokens to an existing vocabulary requires allocating and initializing new rows, for example randomly or from statistics of existing rows. Exact initialization is model/recipe dependent, not necessarily identical for new and original rows. Chat-format training shapes the model’s use of roles through the answer loss; embedding rows change only if trainable. A frozen language model can reuse already learned role embeddings. Nothing is randomly reinitialized per prompt at inference.

Ask: Is USER a separate encoder or a fixed hand-written feature vector?

Answer: No. In a dedicated-token format its ID selects a learned row of the same text embedding table used for ordinary words. The whole language model learns how to use the role cue. Existing embeddings are loaded at inference; newly added tokens need initialized rows and appropriate training.

This is a teaching chat format, not an exact LLaVA serialization. USER and ASST are readable stand-ins. Some models have dedicated single vocabulary tokens for roles; others spell roles using ordinary tokens and separators, so one role label need not equal one token. Image placement, delimiters and tokenization depend on the trained chat template. Keep the same template at training and inference. Positions are zero-based and derived here: twelve visual rows occupy 0–11, USER is 12, the illustrative four word-level question tokens occupy 13–16, and ASST is 17. Real tokenizers may split the question differently, so these exact indices are illustrative. The final supplied ASST state, after causal decoder processing, is sent through the LM head to predict the first answer token. If It is selected, its input embedding is placed at 18 next; it is not supplied in the first forward pass. All positions follow the model’s positional scheme. An additive scheme adds a position vector to input embeddings, while LLaMA uses RoPE rotations on queries and keys inside attention. Do not imply that RoPE is an extra learned token embedding or that it is added to the input. Role markers get no exemption from positions or causal masking. The image encoder has its own earlier position handling; decoder positions apply to the combined input.

Ask: Does a role marker get a position? Do positions restart when the speaker changes?

Answer: Each constituent role token occupies a normal sequence position. In this example USER is at 12 and ASST at 17; positions do not restart at the role boundary. All positions use the decoder’s positional mechanism, including RoPE when that is the architecture.

Read one query row first. The current position may also attend to itself; the simplified row here emphasizes earlier context.

Ask: The assistant’s state can use both the picture and the question.

Answer: Read one query row first. The current position may also attend to itself; the simplified row here emphasizes earlier context.

Compare the same illustrative input under two decoder masks. z1–z3 are projected image vectors from one image, T1–T3 are prompt positions, and A is the assistant-start position. Rows are queries; columns are keys. LEFT: the LLaVA/LLaMA baseline used in this lecture is lower triangular across image and text. z1 reads itself, z2 reads z1 and itself, and z3 reads all three image positions. Qwen2.5-VL also uses a causal decoder, corroborating that this is a common construction, not merely a teaching convention. RIGHT: Gemma 3 is a concrete alternative with bidirectional attention inside the image block. The three additional amber edges let z1 read z2 and z3, and z2 read z3. This diagram abstracts local attention windows, special image markers and multiple images; it is not a complete Gemma 3 layer mask. Text rows are identical in this simplified comparison: they read preceding image and text positions and themselves; neither construction allows future answer tokens to leak into earlier positions. Earlier full self-attention inside ViT has already mixed image information in both cases. That encoder context is not erased by a later causal decoder mask, but neither does it require a bidirectional decoder image block. All supplied image/prompt positions can be processed together in parallel within each layer during prefill; only new answer tokens are generated autoregressively afterwards. Avoid claiming a universal mask for all VLMs.

Ask: Both keep text causal. The difference is which image rows can read each other.

Answer: Compare the same illustrative input under two decoder masks. z1–z3 are projected image vectors from one image, T1–T3 are prompt positions, and A is the assistant-start position. Rows are queries; columns are keys. LEFT: the LLaVA/LLaMA baseline used in this lecture is lower triangular across image and text. z1 reads itself, z2 reads z1 and itself, and z3 reads all three image positions. Qwen2.5-VL also uses a causal decoder, corroborating that this is a common construction, not merely a teaching convention. RIGHT: Gemma 3 is a concrete alternative with bidirectional attention inside the image block. The three additional amber edges let z1 read z2 and z3, and z2 read z3. This diagram abstracts local attention windows, special image markers and multiple images; it is not a complete Gemma 3 layer mask. Text rows are identical in this simplified comparison: they read preceding image and text positions and themselves; neither construction allows future answer tokens to leak into earlier positions. Earlier full self-attention inside ViT has already mixed image information in both cases. That encoder context is not erased by a later causal decoder mask, but neither does it require a bidirectional decoder image block. All supplied image/prompt positions can be processed together in parallel within each layer during prefill; only new answer tokens are generated autoregressively afterwards. Avoid claiming a universal mask for all VLMs.

A simple row-wise linear or MLP projector keeps the patch-row count. The few boxes are a drawn subset, not token selection. LLaVA variants differ in details. Later compression methods deliberately reduce the visual sequence.

Ask: Projected patch vectors occupy earlier positions; they do not become word IDs.

Answer: A simple row-wise linear or MLP projector keeps the patch-row count. The few boxes are a drawn subset, not token selection. LLaVA variants differ in details. Later compression methods deliberately reduce the visual sequence.

Trace the two inputs separately. The purple path embeds the question and tokens generated so far; the answer portion is empty initially. The vision encoder exposes h1, h2, …, hN, a sequence of feature rows. B × N × dV means batch size × retained visual rows per example × features per row; the drawing shows one image. Text states supply Q, image states are projected to K and V. The attention result updates the language states before the vocabulary head. N varies with image resolution, patch size, CLS handling and any compression; 197 is only a conditional example. This schematic shows the standard sequence-to-sequence mechanism, not Flamingo’s exact layer ordering or gates.

Ask: Causal self-attention reads text; cross-attention reads image memory.

Answer: Trace the two inputs separately. The purple path embeds the question and tokens generated so far; the answer portion is empty initially. The vision encoder exposes h1, h2, …, hN, a sequence of feature rows. B × N × dV means batch size × retained visual rows per example × features per row; the drawing shows one image. Text states supply Q, image states are projected to K and V. The attention result updates the language states before the vocabulary head. N varies with image resolution, patch size, CLS handling and any compression; 197 is only a conditional example. This schematic shows the standard sequence-to-sequence mechanism, not Flamingo’s exact layer ordering or gates.

Use the complete source-left, decoder-right architecture from the final diagrams of the earlier lecture. The current target state first uses causal target self-attention and then reads source states. Replace only the source modality on the next slide.

Ask: English source states stayed available while the Hindi prefix grew.

Answer: Use the complete source-left, decoder-right architecture from the final diagrams of the earlier lecture. The current target state first uses causal target self-attention and then reads source states. Replace only the source modality on the next slide.

The text sequence remains separate from visual memory. We have not yet computed a visual message.

Ask: The source changes. The decoder still reads a separate memory.

Answer: The text sequence remains separate from visual memory. We have not yet computed a visual message.

Use the same Newfoundland photograph already familiar from the lecture. Ask students to complete It is a with dog, then Behind it are with trees. Both are authored reference continuations supported by the image, not recorded predictions. The image alone does not specify which answer to give. The language query comes from the current decoder state, which reflects the question and answer prefix. At a fixed cross-attention layer, an unchanged image memory and fixed projection weights give unchanged K and V. A different text context gives a different query and can change its readout. Later-layer language states can also reflect earlier visual reads. No attention heatmap or human-interpretable coordinate is being claimed.

Ask: Why can the same photograph support two different next words?

Answer: The questions and answer prefixes differ. They produce different language states and queries, while the same image memory supplies the same keys and values at this layer. The decoder uses the resulting visual message to continue the requested answer.

Trace the actual photo first, then the concrete question and answer prefix. The 12 rows come from the same 3 × 4 image grid; all twelve h, k and v rows are drawn. A row is a vector, not a named object. ViT contextualization lets every row include information from other patches. W_K and W_V are distinct learned projections of the same image features. K is compared against q; V supplies the content to mix. W_Q maps the contextual language state at the last supplied token a, after causal self-attention, to one query. The state includes preceding question and answer-prefix context; it is not the raw embedding of a. The tokenization, positional treatment and normalization use the decoder’s usual rules; the overview abstracts their internal details. We show one current position and one attention head. The complete decoder can compute queries for every text position. All vectors here are symbolic, not measured from the image. The next slide uses this exact query and these paired rows.

Ask: Does q come directly from the image? Does each key have a paired value?

Answer: q is computed from the contextual state at a, the last supplied token in It is a. Causal self-attention lets that state use the question and earlier answer tokens. The image encoder supplies the rows used for K and V. Row i gives both kᵢ and vᵢ, through different learned projections.

Carry What animal is shown? and It is a through one lookup. The highlighted a is the final supplied token. Select its hidden state after causal self-attention and apply W_Q to obtain q. This is a contextual state with access to the question, It is and a itself, not an isolated word embedding or a pooled state over the text. The image lookup updates that same position. The final state there predicts the next token after a, such as the reference continuation dog; it does not predict a again. A full decoder pass can form queries at every text position, but this next-token example follows only the last position. Normalization and other decoder details are abstracted. Compute a scaled dot product with each of the twelve keys; softmax across those scores gives twelve weights summing to one. Weight alpha_i belongs to value v_i from the same row. Add all twelve weighted value vectors to obtain c. The decoder uses the projected attention output in its residual update, continues its remaining layers, and applies the vocabulary head to the current text state. With multiple heads, their outputs are concatenated and projected first. Dog is an authored reference next word, not a literal word stored in v_i and not a measured generation. The diagram intentionally does not invent numeric weights or a heatmap. For the background question, the same image K,V can be reused but the different incoming text state supplies a different query. We now unpack W_Q, W_K, W_V and the exact arithmetic in the following slides; the following numeric walkthrough keeps all twelve rows and labels its assumed feature values and weights explicitly.

Ask: Which part decides the mixing weights, and where does the word dog appear?

Answer: Comparing q with K determines the scores; softmax gives the weights used to mix V. The result c is a vector. The language head later reads the updated final state at a to score the next token. Dog is the reference continuation after It is a for this illustrated question.

All feature states, tiny dimensions and projection weights are hand-set teaching values, not extracted from the photograph or a model. This is one head and one current language position. All twelve image rows are included. Word labels are readable shorthand for tokenizer pieces; role delimiters are omitted. Encoder rows are contextual, so their information is not restricted to the pictured patch. The question and answer prefix are tokenized and embedded with the decoder’s position handling. The highlighted a is the last available token, not the next token to generate. Its assumed state h_t=[1,2] is not a raw embedding; multiply by W_Q=[[0,1],[0.5,-0.5]] to obtain [1,0]. These are cross-attention query weights, separate from self-attention weights. In a full pass there are queries for every text position; we inspect the final one.

Ask: Use the final supplied text state, after causal self-attention, to make q.

Answer: All feature states, tiny dimensions and projection weights are hand-set teaching values, not extracted from the photograph or a model. This is one head and one current language position. All twelve image rows are included. Word labels are readable shorthand for tokenizer pieces; role delimiters are omitted. Encoder rows are contextual, so their information is not restricted to the pictured patch. The question and answer prefix are tokenized and embedded with the decoder’s position handling. The highlighted a is the last available token, not the next token to generate. Its assumed state h_t=[1,2] is not a raw embedding; multiply by W_Q=[[0,1],[0.5,-0.5]] to obtain [1,0]. These are cross-attention query weights, separate from self-attention weights. In a full pass there are queries for every text position; we inspect the final one.

All feature states, tiny dimensions and projection weights are hand-set teaching values, not extracted from the photograph or a model. This is one head and one current language position. All twelve image rows are included. Word labels are readable shorthand for tokenizer pieces; role delimiters are omitted. Encoder rows are contextual, so their information is not restricted to the pictured patch. Here H_V is 12 by 2, W_K is 2 by 2, and K is 12 by 2. W_K swaps two coordinates solely to make the classroom multiplication visible. The row-6 feature [0,3] becomes key [3,0]. In real models the vision and key widths need not be equal.

Ask: Apply W_K to every encoder row; inspect patch 6 without dropping the other rows.

Answer: All feature states, tiny dimensions and projection weights are hand-set teaching values, not extracted from the photograph or a model. This is one head and one current language position. All twelve image rows are included. Word labels are readable shorthand for tokenizer pieces; role delimiters are omitted. Encoder rows are contextual, so their information is not restricted to the pictured patch. Here H_V is 12 by 2, W_K is 2 by 2, and K is 12 by 2. W_K swaps two coordinates solely to make the classroom multiplication visible. The row-6 feature [0,3] becomes key [3,0]. In real models the vision and key widths need not be equal.

All feature states, tiny dimensions and projection weights are hand-set teaching values, not extracted from the photograph or a model. This is one head and one current language position. All twelve image rows are included. Word labels are readable shorthand for tokenizer pieces; role delimiters are omitted. Encoder rows are contextual, so their information is not restricted to the pictured patch. The same H_V is multiplied by W_V=[[1,0],[1,2]]. Row 6 [0,3] becomes value [3,6], while its key was [3,0]. K and V remain in matching image-position order.

Ask: The key determines the match; its paired value supplies the features to mix.

Answer: All feature states, tiny dimensions and projection weights are hand-set teaching values, not extracted from the photograph or a model. This is one head and one current language position. All twelve image rows are included. Word labels are readable shorthand for tokenizer pieces; role delimiters are omitted. Encoder rows are contextual, so their information is not restricted to the pictured patch. The same H_V is multiplied by W_V=[[1,0],[1,2]]. Row 6 [0,3] becomes value [3,6], while its key was [3,0]. K and V remain in matching image-position order.