CLIP: From Fixed Labels to Language-Defined Vision

An image goes in. We choose the descriptions to compare it with.

The image representation is useful. The classifier chooses from its training labels.

Which supplied description will match the dog photograph best? One matching score for each supplied description, sorted from highest to lowest. The description supplies a comparison vector. Changing the candidates does not train or replace the image encoder. Replace the menu with Newfoundland, pug, and Persian. Run live again; the model weights stay fixed.

Which supplied description will match the dog photograph best? One matching score for each supplied description, sorted from highest to lowest. The description supplies a comparison vector. Changing the candidates does not train or replace the image encoder. Replace the menu with Newfoundland, pug, and Persian. Run live again; the model weights stay fixed.

Can we ask what is inside the cup, rather than just name the cup? A ranking of the four supplied descriptions. Natural language lets us propose attributes, materials and contexts. These are hypotheses to test, not guaranteed abilities. Change only the descriptions to ask about a different property. Each new menu needs a live run.

Can we ask what is inside the cup, rather than just name the cup? A ranking of the four supplied descriptions. Natural language lets us propose attributes, materials and contexts. These are hypotheses to test, not guaranteed abilities. Change only the descriptions to ask about a different property. Each new menu needs a live run.

What will a search for ‘something to drink’ return from this gallery? All 22 gallery images ranked against one text query. Encode the gallery once. A new query needs only a text embedding and dot products. Search always has a nearest item even when nothing matches. Try ‘a laptop’, which is absent from this gallery. A nearest result still exists; decide whether it is relevant.

What will a search for ‘something to drink’ return from this gallery? All 22 gallery images ranked against one text query. Encode the gallery once. A new query needs only a text embedding and dot products. Search always has a nearest item even when nothing matches. Try ‘a laptop’, which is absent from this gallery. A nearest result still exists; decide whether it is relevant.

Which of these four complete sentences best matches the rocket photograph? A score for each candidate sentence. CLIP ranks supplied text. It does not generate captions. A caption generator requires additional machinery. Remove the rocket sentence and rerun. CLIP can only rank the sentences still on the menu.

Which of these four complete sentences best matches the rocket photograph? A score for each candidate sentence. CLIP ranks supplied text. It does not generate captions. A caption generator requires additional machinery. Remove the rocket sentence and rerun. CLIP can only rank the sentences still on the menu.

Which other gallery image will be nearest to the Persian cat photograph? The other 21 gallery images ranked against the query image. The image encoder is useful on its own. Similarity can suggest groups or possible duplicates; it cannot certify that two files are duplicates. Switch the query to the pug or a portrait. No text candidates are needed.

Which other gallery image will be nearest to the Persian cat photograph? The other 21 gallery images ranked against the query image. The image encoder is useful on its own. Similarity can suggest groups or possible duplicates; it cannot certify that two files are duplicates. Switch the query to the pug or a portrait. No text candidates are needed.

Which candidate word best describes the change between the two pictures? One signed cosine between the after − before direction and each candidate word. The person, pose, clothing, framing, and background are intended to stay the same. These are generated pictures. Hair, lighting, and fine details can also change; the pair does not perfectly isolate a hat. Inspired by CodeEmporium’s hat experiment, using our own generated pair. This is an exploratory direction, not a trained change detector. Subtle image differences remain. Cosine is not a probability of the change. Swap before / after and run again. With fixed candidates, every change score should reverse sign.

Which candidate word best describes the change between the two pictures? One signed cosine between the after − before direction and each candidate word. The person, pose, clothing, framing, and background are intended to stay the same. These are generated pictures. Hair, lighting, and fine details can also change; the pair does not perfectly isolate a hat. Inspired by CodeEmporium’s hat experiment, using our own generated pair. This is an exploratory direction, not a trained change detector. Subtle image differences remain. Cosine is not a probability of the change. Swap before / after and run again. With fixed candidates, every change score should reverse sign.

Which word matches the change from the red mug to the blue mug? Signed matches to the change direction, not probabilities that an image contains a blue or red object. The mug shape, handle, position, viewpoint, tabletop, and background are intended to stay the same. Generated reflections, edges, and background details can differ too. The measured direction contains every encoded difference, not only color. Difference directions depend on images, normalization and wording. They are exploratory measurements, not guaranteed semantic algebra. Swap before / after and run again. The scores reverse sign; the highest-ranked word may change.

Which word matches the change from the red mug to the blue mug? Signed matches to the change direction, not probabilities that an image contains a blue or red object. The mug shape, handle, position, viewpoint, tabletop, and background are intended to stay the same. Generated reflections, edges, and background details can differ too. The measured direction contains every encoded difference, not only color. Difference directions depend on images, normalization and wording. They are exploratory measurements, not guaranteed semantic algebra. Swap before / after and run again. The scores reverse sign; the highest-ranked word may change.

Which supplied scene description matches the generated kiln tile? Four scene-description scores for the selected synthetic aerial tile. The search idea transfers to satellite tiles, but synthetic images are not evidence of field performance. Real deployment needs georeferenced imagery, independent labels, and evaluation across regions and resolutions. Compare ‘brick kiln’ with a longer description in a separate live run. Record the images and wording used.

Which supplied scene description matches the generated kiln tile? Four scene-description scores for the selected synthetic aerial tile. The search idea transfers to satellite tiles, but synthetic images are not evidence of field performance. Real deployment needs georeferenced imagery, independent labels, and evaluation across regions and resolutions. Compare ‘brick kiln’ with a longer description in a separate live run. Record the images and wording used.

Which word matches the change between the two kiln scenes? A ranked match between the image-change direction and four words. The kiln, brick stacks, camera position, and surrounding landscape are intended to stay the same. The chimney brings a shadow; generated fine details also vary. This pair is not a verified physical intervention or a trained kiln detector. Synthetic paired scenes test a hypothesis. Shadows and fine details also change; this is not a causal proof or a validated kiln detector. Swap the order and rerun. Inspect the pictures for changed shadows and background details.

Which word matches the change between the two kiln scenes? A ranked match between the image-change direction and four words. The kiln, brick stacks, camera position, and surrounding landscape are intended to stay the same. The chimney brings a shadow; generated fine details also vary. This pair is not a verified physical intervention or a trained kiln detector. Synthetic paired scenes test a hypothesis. Shadows and fine details also change; this is not a causal proof or a validated kiln detector. Swap the order and rerun. Inspect the pictures for changed shadows and background details.

This is an input-image intervention, not an edit to an internal concept bottleneck. The generated edit can change shape, texture and fine detail as well as stripes. It suggests sensitivity to the edit; one pair does not establish that stripes alone caused the prediction change. Softmax shares are introduced only after the loss walkthrough.

This is an input-image intervention, not an edit to an internal concept bottleneck. The generated edit can change shape, texture and fine detail as well as stripes. It suggests sensitivity to the edit; one pair does not establish that stripes alone caused the prediction change. Softmax shares are introduced only after the loss walkthrough.

Can CLIP match this public radiograph to an imaging modality? A ranking of image-type descriptions, not a diagnosis. This public radiograph supports a modality-matching lesson, not diagnosis. General CLIP scores are not clinical evidence; medical tasks need domain-specific validation. Keep the distinction clear: identifying an image type is a different task from diagnosing disease.

Can CLIP match this public radiograph to an imaging modality? A ranking of image-type descriptions, not a diagnosis. This public radiograph supports a modality-matching lesson, not diagnosis. General CLIP scores are not clinical evidence; medical tasks need domain-specific validation. Keep the distinction clear: identifying an image type is a different task from diagnosing disease.

Will the added device make ‘pacemaker’ the highest-scoring word? Four exploratory word scores for a synthetic change; no clinical finding is inferred. The chest outline, viewpoint, and radiograph appearance are intended to stay the same. Both images are generated; anatomy and device placement are approximate, and other pixels differ. This is not a patient before/after study. Both radiographs are generated illustrations. Anatomy and device placement are approximate. This measures concept matching on synthetic pictures, not diagnostic or clinical performance. Try fuller descriptions in a separate run. Record the original failure alongside any improvement.

Will the added device make ‘pacemaker’ the highest-scoring word? Four exploratory word scores for a synthetic change; no clinical finding is inferred. The chest outline, viewpoint, and radiograph appearance are intended to stay the same. Both images are generated; anatomy and device placement are approximate, and other pixels differ. This is not a patient before/after study. Both radiographs are generated illustrations. Anatomy and device placement are approximate. This measures concept matching on synthetic pictures, not diagnostic or clinical performance. Try fuller descriptions in a separate run. Record the original failure alongside any improvement.

Can the dog concept still match when the image is a pencil sketch? One similarity per description for a single image; there is no subtraction in this example. This is a small domain-shift experiment. A success here does not establish robustness to every drawing style. Open Style change to ask a different question: what changed between the two images?

Can the dog concept still match when the image is a pencil sketch? One similarity per description for a single image; there is no subtraction in this example. This is a small domain-shift experiment. A success here does not establish robustness to every drawing style. Open Style change to ask a different question: what changed between the two images?

Does the change direction emphasize pencil sketch, photograph, dog, or cat? A signed match to the style-change direction, distinct from recognizing the dog in either image. The dog identity, pose, and composition are intended to remain similar. Color, texture, background, and small shapes all change. This direction does not isolate a single style coordinate. This subtraction includes background and rendering changes. Compare original image-to-text matching with the difference direction; a successful guess does not isolate a causal concept. Swap the order and rerun. Then use Photo → sketch to classify each image separately.

Does the change direction emphasize pencil sketch, photograph, dog, or cat? A signed match to the style-change direction, distinct from recognizing the dog in either image. The dog identity, pose, and composition are intended to remain similar. Color, texture, background, and small shapes all change. This direction does not isolate a single style coordinate. This subtraction includes background and rendering changes. Compare original image-to-text matching with the difference direction; a successful guess does not isolate a causal concept. Swap the order and rerun. Then use Photo → sketch to classify each image separately.

What will win when none of the descriptions says ‘dog’? Scores and relative shares across the supplied menu, even if every option is wrong. Softmax distributes 100% across the choices you provide. It is not calibrated confidence that the best label is true. Add ‘a photo of a dog’ and rerun. The old pairwise cosines stay fixed, while their relative shares change.

What will win when none of the descriptions says ‘dog’? Scores and relative shares across the supplied menu, even if every option is wrong. Softmax distributes 100% across the choices you provide. It is not calibrated confidence that the best label is true. Add ‘a photo of a dog’ and rerun. The old pairwise cosines stay fixed, while their relative shares change.

Will ‘a photo without a cat’ score much lower than ‘a photo of a cat’? A similarity for each complete phrase; no rule explicitly implements logical negation. A flexible vocabulary does not guarantee precise compositional reasoning. Report what this checkpoint does on these examples, not universal claims. Try another phrasing while keeping the photograph fixed. Save failures as well as successes.

Will ‘a photo without a cat’ score much lower than ‘a photo of a cat’? A similarity for each complete phrase; no rule explicitly implements logical negation. A flexible vocabulary does not guarantee precise compositional reasoning. Report what this checkpoint does on these examples, not universal claims. Try another phrasing while keeping the photograph fixed. Save failures as well as successes.

All fifteen original lab experiments are back in the opening. Keep the medical-device failure and the wrong-menu example visible. The next section returns to the two encoders; the complete loss is worked through later.

The earlier ImageNet classifier has 1,000 output categories. Each row of its PyTorch head.weight tensor stores 192 learned weights; head.bias stores one bias for each row. We show actual checkpoint rows for Newfoundland (dog), Persian cat and macaw (bird), reordered for this illustration. The dog/cat/bird subscripts are shorthand for these three specific categories, not broad ImageNet labels. Only the first three coordinates are printed; all 192 enter the dot product. The names label the output rows and are not sent through a text encoder. Labelled training learned both the ViT and these class parameters. During prediction, only the computed image features change. These are affine scores, not cosine similarities. Typing a new class name does not supply another row of learned weights; the following slide asks how language could provide the comparison vector instead.

The previous classifier compared image features h with a stored weight vector for each fixed category. That vector is what we mean by the class-side comparison vector: it determines how the image is scored against that category. CLIP supplies this side of the comparison by encoding a description into v, so our candidate list is not limited to the earlier 1,000 stored classifier rows. The image is encoded by CLIP’s own image encoder to u; we do not attach text vectors to the earlier 192-dimensional ViT features. Both displayed CLIP encoders include their learned projection and unit normalization, expanded in the following slides. Their final vectors have 512 coordinates and were trained together to be comparable. The diagram shows raw cosine u dot v, while the earlier classifier uses an affine score with a bias. This is a bridge between the roles of the vectors, not equality of the two models or their spaces. A new description changes the computed text vector, not the pretrained parameters; it also does not guarantee a reliable match.

Use the final CLS state after the ViT blocks and LayerNorm.

A learned 768 × 512 matrix maps the image summary into the comparison space.

Divide by the vector’s length. The resulting image vector has length 1.

Show the complete route first: each input passes through its own encoder, learned projection and unit normalization, then the two vectors meet at a cosine score. The purple highlight marks the text Transformer and its final EOT readout. The next slide enlarges this part; the following slides walk through text projection and normalization.

Connect to the sentence-readout diagram in the earlier From Attention to Applications lecture, referred to here as Beyond Attention. There we selected the updated CLS state after full self-attention. Here we use the same readout idea, but select the final EOT state after causal self-attention; these are analogous roles, not identical tokens or vectors. SOT means start of text (<|startoftext|>); EOT means end of text (<|endoftext|>). Both are special input tokens added around the description, not start or end of an individual word. EOT follows all description tokens; padding, omitted here, can follow EOT. For example, “photo” can attend to SOT, “a” and itself, but not the later “dog” token. EOT can attend to all preceding tokens and itself. SOT cannot collect the later words under this causal mask. The input EOT embedding is not the final contextual state: the Transformer computes that state. LN is the final LayerNorm; h_T has 512 features in ViT-B/32, before the learned text projection and unit normalization on the next slides. This is an encoder-like use of the Transformer: produce one representation of the supplied text. It does not change the causal attention mask. We train with CLIP’s contrastive loss; there is no next-token prediction head or next-token training loss. The paper says masked attention preserved the option of adding language modelling as an auxiliary objective, but leaves that experiment to future work. The image ViT uses full self-attention across its class token and patches.

A learned 512 × 512 projection can change direction without changing width.

We now have one image vector and one text vector, each with 512 coordinates and unit length.

The dot product depends on both lengths and the angle between the vectors.

Normalization makes the dot product equal to cosine similarity.

This is geometry. Training must make this geometry useful.

This is geometry. Training must make this geometry useful.

This is geometry. Training must make this geometry useful.

The branches do not exchange tokens. They meet after encoding, at the dot product.

Same width permits comparison. Training gives the comparison meaning.

The model has no useful pairing information merely because both outputs have width 512.

Learn a geometry in which observed pairs beat competing pairs.

The matching information is carried by the learned geometry.

Return to the toy dog image and the descriptions used in the alignment examples. Suppose both branches map every input to the same unit vector. The dot product of that vector with itself is 1, so every image–caption comparison ties at cosine 1. In the table, each row names one supplied caption; the number on its right is the cosine between that caption vector and this one image’s vector. The highlighted row identifies the supplied training pair, not a model-selected winner. These are hypothetical values, not measured CLIP outputs or probabilities. We cannot identify the observed partner from tied scores. This failure is called representation collapse. The next slide motivates comparison with competitors; we introduce batches and the full image-by-caption matrix afterward.

Contrastive training compares the observed pairing with alternative pairings.

The original CLIP paper describes a query-filtered collection of 400M image–text pairs. Our 12 records are authored teaching examples. The illustrations, supplied captions and chosen vectors are the same as in the tutorial. They are teaching examples, not measured CLIP embeddings. Associated web text can be noisy, incomplete, duplicated or only loosely related.

For this walkthrough, take cat, dog and car as the first batch.

Keep image 1 with text 1, image 2 with text 2, and image 3 with text 3.

I denotes an image and T its supplied text. The matching indices identify the sampled pair records. An off-diagonal caption is treated as a competitor here; it need not be a false description in general.

The toy cat illustration identifies the same record throughout. Its supplied text is “a photo of a cat”. We choose h and W to explain the projection; no trained image model produced these four feature values. The multiplication reproduces the interactive’s existing raw cat vector exactly. Use row vectors here: h is 1×4, W is 4×3 and z is 1×3. Real CLIP ViT-B/32 uses 768 image features projected to 512 coordinates. Projection is linear and learned; unit normalization is a separate operation. The source interactive begins at the projected vectors. The overview previews both paths; the following slides expand features, projection and normalization. Normalization changes vector length, not the number of coordinates. This restores the old worksheet-notation link.

The toy cat illustration identifies the same record throughout. Its supplied text is “a photo of a cat”. We choose h and W to explain the projection; no trained image model produced these four feature values. The multiplication reproduces the interactive’s existing raw cat vector exactly. Use row vectors here: h is 1×4, W is 4×3 and z is 1×3. Real CLIP ViT-B/32 uses 768 image features projected to 512 coordinates. Projection is linear and learned; unit normalization is a separate operation. The source interactive begins at the projected vectors.

The toy cat illustration identifies the same record throughout. Its supplied text is “a photo of a cat”. We choose h and W to explain the projection; no trained image model produced these four feature values. The multiplication reproduces the interactive’s existing raw cat vector exactly. Use row vectors here: h is 1×4, W is 4×3 and z is 1×3. Real CLIP ViT-B/32 uses 768 image features projected to 512 coordinates. Projection is linear and learned; unit normalization is a separate operation. The source interactive begins at the projected vectors.

The toy cat illustration identifies the same record throughout. Its supplied text is “a photo of a cat”. We choose h and W to explain the projection; no trained image model produced these four feature values. The multiplication reproduces the interactive’s existing raw cat vector exactly. Use row vectors here: h is 1×4, W is 4×3 and z is 1×3. Real CLIP ViT-B/32 uses 768 image features projected to 512 coordinates. Projection is linear and learned; unit normalization is a separate operation. The source interactive begins at the projected vectors.

The toy cat illustration identifies the same record throughout. Its supplied text is “a photo of a cat”. We choose h and W to explain the projection; no trained image model produced these four feature values. The multiplication reproduces the interactive’s existing raw cat vector exactly. Use row vectors here: h is 1×4, W is 4×3 and z is 1×3. Real CLIP ViT-B/32 uses 768 image features projected to 512 coordinates. Projection is linear and learned; unit normalization is a separate operation. The source interactive begins at the projected vectors.

The toy cat illustration identifies the same record throughout. Its supplied text is “a photo of a cat”. We choose h and W to explain the projection; no trained image model produced these four feature values. The multiplication reproduces the interactive’s existing raw cat vector exactly. Use row vectors here: h is 1×4, W is 4×3 and z is 1×3. Real CLIP ViT-B/32 uses 768 image features projected to 512 coordinates. Projection is linear and learned; unit normalization is a separate operation. The source interactive begins at the projected vectors.

The text branch reads the final contextual EOT state after LayerNorm, applies its own learned projection, and normalizes the resulting vector. The displayed raw text vector and unit vector are the original tutorial’s chosen values, not measured embeddings. The projection uses W_T for text and W_I for images. The stacked coordinate lists are not matrix-orientation notation; the multiplication uses row vectors.

These are the source interactive’s chosen text outputs after projection. Divide by their Euclidean norm; no extra model is run.

The toy cat illustration identifies the same record throughout. Its supplied text is “a photo of a cat”. We choose h and W to explain the projection; no trained image model produced these four feature values. The multiplication reproduces the interactive’s existing raw cat vector exactly. Use row vectors here: h is 1×4, W is 4×3 and z is 1×3. Real CLIP ViT-B/32 uses 768 image features projected to 512 coordinates. Projection is linear and learned; unit normalization is a separate operation. The source interactive begins at the projected vectors.

These are the original interactive’s six chosen post-projection outputs. Only the cat-image projection was expanded as a worked hW calculation; the other raw vectors remain the provided toy outputs.

The geometry is the source tutorial’s orthographic view of the actual chosen 3D unit vectors. It is not a t-SNE/PCA projection of measured CLIP features. The source geometry labels I/T are changed to the lecture’s u/v; positions and vectors are unchanged.

Row 1 is the cat image. Its three columns are coordinates, not classes.

Row 1 is “a photo of a cat”. Keep it aligned with image row 1.

Transpose moves the numbers. Text 1’s coordinates now run down column 1.

The tutorial labels normalized matrices EI and ET and cosine scores S. The lecture keeps U, V and C introduced earlier. These names refer to exactly the same numeric arrays; no values change.

The same coordinate positions meet: first with first, second with second, third with third.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.

Rounded coordinates are printed; the displayed sum is computed with full-precision normalized vectors. The figures use the interactive’s own SVG matrix renderer, keeping every cell in the same position.