Part 2 opened with a sentence and a blank. This part opens with a picture and a blank. The four words how many mugs ? are the same for every picture, so whatever fills the blank must come from the image. CLIP, from Vision III, can only compare the picture with captions we write for it. Here a language decoder, the same one-head block as Part 2, reads sixteen patch rows and the question, and predicts the next token. Switch the scene under the bars: the answer moves with the mugs, before you see any machinery.

The whole part fits in one picture. The scene is cut into sixteen 2×2 patches. Vision I's encoder, with its trained attention frozen, turns them into sixteen rows of four named numbers. A connector maps each row to the decoder's width of three. The five words of the question become five more rows through a lookup table. All twenty-one rows enter the block from Part 2: queries, keys, values, one softmax per row, a message, an update, a residual addition. The question mark's updated row goes into the vocabulary head, and the tallest bar is two. Photograph this figure; every later section is a close-up of one of its parts.

Vision I ended with sixteen updated patch rows on the axes brightness, contrast, row and col; we keep them exactly, drop the CLS row, and freeze them. Notice that row and col are no longer plain positions: the encoder's update landed there, so those two numbers also carry the brightness a patch read from its neighbours. The connector multiplies every row by one 4×3 matrix and adds a bias. Its three output axes were learned, so we name each after the pixel quantity it tracks across the sixteen patches; the first tracks brightness almost exactly. The five words get rows from a lookup table plus a position vector.

All sixteen rows, with the thumbnail of each patch. The CLS row is dropped: the decoder will read the patches themselves.

One patch worked through the connector, one output axis per line:

Part 2 used a causal mask so that no token could read the future. With image rows in front, the mask has two rules. Image rows read only image rows: they are context, not part of the sentence, and they must never carry a later answer word backwards. Text rows read every image row and the words up to themselves. The picture below shows the 21×21 pattern. There is a second way to connect an image to a decoder, the cross-attention of Part 4, where the text supplies queries and a separate visual memory supplies keys and values; our numbers use the shared prefix, and the same mask.

In Part 2 the receiving token was bank. Here it is the question mark, the last known token, because the unknown answer cannot supply a query. Its query is compared with all twenty-one keys and a softmax turns the scores into weights. Instead of a column of numbers, paint them on the scene: each patch is tinted by its weight. The four mug patches take almost everything; the text rows get nothing. The worksheet behind the overlay is the same q·k arithmetic as Part 2, and the mixed values, the update and the vocabulary head follow in the same order. Every number here is scene A at the fitted checkpoint.

The logit of the winner, written out: the updated row times the two column of $W_{\mathrm{vocab}}$, plus its bias.

One prediction is not yet an answer. Generation repeats the same forward pass with a longer prefix: choose the most probable token, append it as a new row, and let that row ask the next question. Step through it below. The question mark predicts two; the appended two row forms its own query, reads the scene again, and predicts the end marker, which stops generation. The parameters never change and the sixteen image rows never change; only the last query does. Look at the second overlay: the end marker's decision reads the table and the book, not the mugs.

This is the payoff of the series. Keep the five words fixed and swap the picture. In scene A the first answer token reads both mugs and says two; in scene B it reads the one mug that is left and says one. Then a scene the model never saw: the single mug moved to the right. The model reads the moved mug and answers two. It did not learn to count; it learned that a bright mug on the right means two, which is exactly the question Vision I's frozen encoder was trained to answer. Two training scenes cannot tell those rules apart.

Training uses teacher forcing: the correct answer is placed after the question, the mask keeps each row from seeing later tokens, and only the answer rows are scored. Here that is two targets: the question mark should predict two, and the two row should predict the end marker. The run below fits scenes A and B in 73 Adam steps; the overlay of what the question mark reads goes from an even smear to the four mug patches. Then one more step on scene A alone: its loss falls, scene B's loss rises. Keep that failure; it is the reason for batches and held-out checks.

Grounding has two sides: the answer must follow the image, and the image must contain the evidence. The first probe changes one word of the question, mugs to book. The toy answers two again, with an identical overlay, because in this one-block decoder only the question mark's row asks, and that row does not depend on the word before it. The second probe cannot even be run: how heavy is the mug? has no answer in sixty-four brightness values, and no word for it in the vocabulary. When the input carries no evidence, a confident answer is a guess, however fluent.

Explain this part without the page. In a sentence: the image rows are context, and the answer is the next token. As a chain: pixels, frozen rows, connector, stacked with the words, one block, the head. As equations: the four lines below, with their shapes. Then try the questions; each answer carries a number computed by the toy, including a counterfactual with the connector switched off. The last frame puts Part 1's first prediction, the letter after a a b, beside this part's last one. The shapes match: a prefix in, a distribution out.