00 · LANGUAGE CHANGES THE TARGET

LANGUAGE-GROUNDED VISION

How Words Become Boxes, Regions, and Masks

HOW DOES LANGUAGE CHANGE WHICH PART OF THE IMAGE WE PREDICT?

ES 667 · Nipun Batra · 55-minute route

Fixed park illustration with staged reference annotation

Same image. Change language. Change the target region.

ES 667 · Nipun Batra · authored teaching diagrams01 / 56

The fixed park scene is an authored illustration. All main teaching overlays are reference annotations; recorded model evidence is identified separately. The child now visibly kicks the red ball; this same drawing is used throughout. Ten dividers include this title slide.

Ask:

Reference answer:

00 · LANGUAGE CHANGES THE TARGET

A detector already gives us locations

Fixed park illustration with staged reference annotation

“dog”

I⟼{(b^i,ci,si)}i=1NI\longmapsto\{(\hat{\mathbf b}_i,c_i,s_i)\}_{i=1}^{N}

Box · class label · score
Familiar fixed-class detector

Where is already part of detection. Which target is still constrained by the label set.

ES 667 · Nipun Batra · authored teaching diagrams02 / 56

Recall the recent detector lecture. These are reference boxes, not measured predictions. Do not assign an invented score to the scene.

Ask: What does the image-level label “dog” leave out?

Reference answer: Location and the number of instances.

00 · LANGUAGE CHANGES THE TARGET

“Person” does not identify one person

Three people labelled A, B, C wearing blue, red and green shirtsABC

“person”

A, B and C all match.

“person wearing red”

B · the attribute selects an instance

A class name is sometimes not enough.

ES 667 · Nipun Batra · authored teaching diagrams03 / 56

Preserved three-person illustration. Point to all three before adding red. Ask for the selected instance before revealing B.

Ask: What information did “wearing red” add?

Reference answer: An attribute that distinguishes instances of the same class.

00 · LANGUAGE CHANGES THE TARGET

Change the phrase; change the target

Fixed scene: query reveals first the ball box, then the dog box

“red ball”

Attribute selects the red ball.

“ball beside the dog”

Relation selects that same ball.

“dog beside the red ball”

Now the requested object is the dog.

The head noun names the target; the relation constrains it.

ES 667 · Nipun Batra · authored teaching diagrams04 / 56

Two boxes remain at the completed state to compare target noun roles. During teaching first point at the ball, then dog. Nearness is obvious in this authored scene. This example alone cannot establish model relational understanding.

Ask: Which object should be returned for “dog beside the red ball”?

Reference answer: The dog. The ball is the reference object, not the requested target.

00 · LANGUAGE CHANGES THE TARGET

The words can ask for unfamiliar evidence

Schematic brick kiln footprint and central chimney, not a satellite observationSchematic aerial view · no model inference

“find the kiln”

“find its chimney”

Same domain. Different spatial target.

Schematic aerial view.
Not a satellite observation.

An open vocabulary does not remove domain shift or missing visual detail.

ES 667 · Nipun Batra · authored teaching diagrams05 / 56

Preserve kiln opening but compress it. The oval kiln and central chimney illustrate different regions. Recognizing a domain term still requires suitable image evidence and training.

Ask: Would a text encoder make an unresolved chimney visible?

Reference answer: No. The image resolution and domain evidence still limit localization.

01 · CLIP, BUT SPATIAL

CLIP, BUT SPATIAL

01

CLIP matches a description. How do we answer WHERE?

Fixed park illustration with staged reference annotation

CLIP matches a description. How do we answer WHERE?

ES 667 · Nipun Batra · authored teaching diagrams06 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

01 · CLIP, BUT SPATIAL

IF is a different question from WHERE

Architecture diagramGlobal image vector“a dog in a park”Text vectorWhole-image similarity

One global image vector does not specify a box.

Connect to the CLIP matrix intuition. Global matching can support image retrieval. Localizing requires retaining spatial information and learning a geometric output.

Ask: What has pooling removed?

Reference answer: Explicit per-location output structure.

01 · CLIP, BUT SPATIAL

Replace the fixed label representation

Architecture diagramLocal visual vectorLearned class weightsFixed class scores

Supply the class representation through a text encoder.

sij=ri⊤tjs_{ij}=r_i^{\top}t_j

Text changes which semantic columns we ask the detector to score.

ES 667 · Nipun Batra · authored teaching diagrams08 / 56

The first row recalls a conventional linear class head. Then replace learned class weight vectors by encoded descriptions. Actual open-vocabulary models need local feature alignment and detection training.

Ask: Does changing the label list alone teach box geometry?

Reference answer: No. Geometry requires spatial features and a trained box predictor.

01 · CLIP, BUT SPATIAL

One dot product becomes one matrix cell

r1=[0.6,0.8],tdog=[1,0]r_1=[0.6,0.8],\qquad t_{\mathrm{dog}}=[1,0]
s1,dog=r1⊤tdogs_{1,\mathrm{dog}}=r_1^\top t_{\mathrm{dog}}
=0.6×1+0.8×0=0.6\times1+0.8\times0
=0.6=0.6
row 1 · dog query
0.6

Normalized, illustrative vectors. These are not measured park features.

The scalar 0.6 answers one region–text comparison.

ES 667 · Nipun Batra · authored teaching diagrams09 / 56

Go slowly: multiply the first components, then second, then add. Step 5 moves the scalar into the amber outlined cell. The vectors both have unit norm.

Ask: What does 0.6 refer to?

Reference answer: Region 1 compared with the supplied dog vector; not a probability.

01 · CLIP, BUT SPATIAL

Build the matrix from comparisons

tball=[0,1]t_{\mathrm{ball}}=[0,1]
s1,ball=0.6(0)+0.8(1)=0.8s_{1,\mathrm{ball}}=0.6(0)+0.8(1)=0.8
r2=[0.8,0.6]⇒[0.8,0.6]r_2=[0.8,0.6]\Rightarrow [0.8,0.6]
r3=[−1,0]⇒[−1,0]r_3=[-1,0]\Rightarrow [-1,0]
dog
ball
r₁
0.6
0.8
r₂
0.8
0.6
r₃
−1.0
0.0
R∈RN×d,T∈RM×d,S=RT⊤R\in\mathbb R^{N\times d},\quad T\in\mathbb R^{M\times d},\quad S=RT^\top

ROW: which text fits this region? COLUMN: which region fits this text?

Rows index candidate regions. Columns index supplied text queries.

ES 667 · Nipun Batra · authored teaching diagrams10 / 56

The full 3×2 matrix appears only after several cells. The r vectors are toy candidates, not semantic labels for the park. No claim is made that row 1 is a dog. Ask row and column interpretations separately.

Ask: For dog, which toy region scores highest?

Reference answer: Column dog: r₂ at 0.8. Row r₁ instead prefers ball at 0.8.

01 · CLIP, BUT SPATIAL

Similarity, logit and probability are distinct

Architecture diagramCosine similarityDivide by temperatureSoftmax over candidates
sij=ri⊤tj,zij=sij/τ,p(j∣i)=ezij∑keziks_{ij}=r_i^\top t_j,\quad z_{ij}=s_{ij}/\tau,\quad p(j\mid i)=\frac{e^{z_{ij}}}{\sum_k e^{z_{ik}}}

This softmax is a teaching example. Detectors may instead use independent sigmoid scores.

A cosine score is not automatically a calibrated confidence.

ES 667 · Nipun Batra · authored teaching diagrams11 / 56

No new contrastive-loss derivation. Normalization choices matter. A probability over a supplied list changes if the list changes; it does not prove the correct object exists.

Ask: What happens when we add another text candidate?

Reference answer: The cosine values can remain fixed while the softmax probabilities change.

02 · HOW DOES TEXT BECOME A DETECTION?

HOW DOES TEXT BECOME A DETECTION?

02

Keep the spatial features. Add geometry. Match text.

Fixed park illustration with staged reference annotation

Keep the spatial features. Add geometry. Match text.

ES 667 · Nipun Batra · authored teaching diagrams12 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

02 · HOW DOES TEXT BECOME A DETECTION?

Derive a text-defined detector

Spatial image features split into box prediction and text matchingVisual backboneSpatial featuresone vector per regionBox head → boxesText queriesText encoderText vectorsRegion–text scoreslocal vectorsGrounded detectionsboxes + text scores

CLIP-like semantics + spatial features + box prediction.

ES 667 · Nipun Batra · authored teaching diagrams13 / 56

Six reveals: visual backbone; spatial features; box head; text embeddings; local text matching; boxes combined with scores. Do not name the exemplar before the mechanism is complete. Scores and geometry are different predictions.

Ask: Where does language enter this design?

Reference answer: Through text embeddings compared with local visual features.

02 · HOW DOES TEXT BECOME A DETECTION?

Example: OWL-ViT-style detection

Spatial image features split into box prediction and text matchingVisual backboneSpatial featuresone vector per regionBox head → boxesText queriesText encoderText vectorsRegion–text scoreslocal vectorsGrounded detectionsboxes + text scores

The local features are trained for detection; this is more than global CLIP retrieval.

Name OWL-ViT now. Simplification: image-token outputs, projections and detection heads stand in for implementation details. OWL-ViT transfers image-text pretraining and fine-tunes localization. Open vocabulary is a capability with limits, not an unlimited guarantee.

Ask: Which branch would fail if box training were missing?

Reference answer: The geometry branch would not have learned to predict useful enclosing boxes.

03 · FROM CATEGORY TO REFERRING EXPRESSION

FROM CATEGORY TO REFERRING EXPRESSION

03

Which particular instance does the phrase mean?

Fixed park illustration with staged reference annotation

Which particular instance does the phrase mean?

ES 667 · Nipun Batra · authored teaching diagrams15 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

03 · FROM CATEGORY TO REFERRING EXPRESSION

A description can add constraints

Fixed park illustration with staged reference annotation

“dog”

“brown dog”

“dog beside the red ball”

“dog beside the child”

CATEGORY → ATTRIBUTE → RELATION

Phrase grounding links language to the region it refers to.

All four phrases refer to the same dog here. That does not test whether a model uses relations. The next slide adds same-class alternatives. In a single-referent setting this is referring expression comprehension.

Ask: Does success on this one-dog image prove relation understanding?

Reference answer: No. The model could return the only dog for every prompt.

03 · FROM CATEGORY TO REFERRING EXPRESSION

Make the alternatives plausible

Three authored dogs: standing on the left, sitting in the centre, lying down on the rightA · standingB · sittingC · lying down

“dog”

Three valid instances.

“the sitting dog in the middle”

B · posture and position select it.

A useful grounding test includes competing instances of the same class.

ES 667 · Nipun Batra · authored teaching diagrams17 / 56

Retain the existing three-dog asset. Use it to distinguish detecting a category from resolving a referring expression. These are reference annotations, not a benchmark.

Ask: Why is this a stronger target-selection test?

Reference answer: Several regions satisfy the noun, so additional words must disambiguate.

04 · LANGUAGE-GUIDED OBJECT QUERIES

LANGUAGE-GUIDED OBJECT QUERIES

04

What if language could affect which queries are useful?

Fixed park illustration with staged reference annotation

What if language could affect which queries are useful?

ES 667 · Nipun Batra · authored teaching diagrams19 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

04 · LANGUAGE-GUIDED OBJECT QUERIES

Recall DETR: queries read the image

Architecture diagramLearned object queriesQueries QImage featuresKeys K, values VCross-attentionupdated queries → boxes

A query is a learned request for information, not a preassigned object name.

Recall learned object queries, visual K,V and per-query predictions. The diagram omits query self-attention, positional encodings and repeated layers.

Ask: Do the learned slots begin as “dog query” and “ball query”?

Reference answer: No. Explicit semantic slot labels would be misleading.

04 · LANGUAGE-GUIDED OBJECT QUERIES

The same operation, a spatial question

Architecture diagramDecoder query QEncoder memory K,VUpdated representation

Earlier: generate the next word.
Here: locate what a phrase refers to.

Attention⁡(Q,K,V)=softmax⁡(QK⊤/dk)V\operatorname{Attention}(Q,K,V)=\operatorname{softmax}(QK^\top/\sqrt{d_k})V

Beyond Attention → object queries reading visual and multimodal memories.

ES 667 · Nipun Batra · authored teaching diagrams21 / 56

Make the connection explicit rather than treating multimodal attention as a new primitive. Different sublayers may read visual and textual memories; they need not concatenate everything into one matrix.

Ask: What changes while the operation stays the same?

Reference answer: The query source, memory contents, training targets and output head.

04 · LANGUAGE-GUIDED OBJECT QUERIES

One query mixes three values

q=[1,0],K=[2002−20]q=[1,0],\quad K=\begin{bmatrix}2&0\\0&2\\-2&0\end{bmatrix}
qK⊤=[2,0,−2]qK^\top=[2,0,-2]
z=qK⊤/2=[1.414,0,−1.414]z=qK^\top/\sqrt2=[1.414,0,-1.414]

Key locations: dog · red ball · background

α=softmax⁡(z)≈[0.768,0.187,0.045]\alpha=\operatorname{softmax}(z)\approx[0.768,0.187,0.045]
V=[100100]V=\begin{bmatrix}1&0\\0&1\\0&0\end{bmatrix}
αV≈[0.768,0.187]\alpha V\approx[0.768,0.187]

ILLUSTRATIVE ATTENTION WORKSHEET · spatial labels locate toy keys, not hidden semantics.

Attention produces a value mixture; a downstream head still has to predict geometry.

ES 667 · Nipun Batra · authored teaching diagrams22 / 56

Calculate raw dot products first, then divide by sqrt(2), then softmax, then weighted values. All vectors are invented for arithmetic, not measured activations. Explain that value coordinates have no assigned dog/red-ball meaning in a real model.

Ask: Is the attention distribution already a box?

Reference answer: No. It is a weighting used to update a representation.

04 · LANGUAGE-GUIDED OBJECT QUERIES

Let text influence feature selection

Seven-stage language-guided query detector; visual and text memories reach decoderImage → backboneText → encoderFeature enhancer: visual features ↔ text featuresLanguage-guided query selectionCross-modal DETR decodervisualmemorytextmemoryBoxes + phrase scores

Language can influence features, selected queries and decoder updates.

ES 667 · Nipun Batra · authored teaching diagrams23 / 56

Seven reveals: image branch, text branch, feature enhancer, language-guided query selection, decoder, both memory side paths, then box and phrase-score outputs. Query selection retains language-relevant visual positions. No chat interface is implied.

Ask: Which part changes the initial queries supplied to the decoder?

Reference answer: Language-guided query selection.

04 · LANGUAGE-GUIDED OBJECT QUERIES

This is the Grounding DINO design

Seven-stage language-guided query detector; visual and text memories reach decoderImage → backboneText → encoderFeature enhancer: visual features ↔ text featuresLanguage-guided query selectionCross-modal DETR decodervisualmemorytextmemoryBoxes + phrase scores

A language-conditioned detector, with boxes and text alignment as outputs.

Only now attach the name Grounding DINO. Distinguish it from DINO self-supervised visual learning and from a generative chat VLM. The architecture supports language conditioning; robust relational understanding must still be evaluated.

Ask: Why can a longer phrase still fail?

Reference answer: The model may latch onto nouns, confuse relations or select the wrong instance.

05 · TEXT → BOX → MASK

TEXT → BOX → MASK

05

Language selects a region. How do we recover its pixels?

Fixed park illustration with staged reference annotation

Language selects a region. How do we recover its pixels?

ES 667 · Nipun Batra · authored teaching diagrams25 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

05 · TEXT → BOX → MASK

Compose a grounder with a segmenter

Grounding detector supplies box; SAM uses box prompt and cached image featuresdog beside red ballGrounding modelGrounded boxBox prompt encoderSAM image encodercache once per imageImage embeddingMask decoder

Language selects the region. SAM recovers its precise shape.

Six reveals correspond to the requested six frames: query and image; grounder and box; box prompt; cached SAM image embedding; mask decoder; reference mask. Use the same image in both models. Original released SAM accepts spatial prompts; text is handled upstream.

Ask: What passes from the grounder to SAM?

Reference answer: A spatial box after coordinate conversion.

05 · TEXT → BOX → MASK

The user asks for a mask

Fixed park illustration with staged reference annotation

“segment the dog beside the red ball”

One requested region.
One pixel mask.

Intermediate boxes can disappear from the user’s view.

A clean mask can still belong to the wrong instance.

ES 667 · Nipun Batra · authored teaching diagrams27 / 56

Show the final text-to-mask view without intermediate box overlays. Grounding and shape quality are separable. The displayed mask is an authored reference. Coordinate bookkeeping is preserved in backup.

Ask: Can the segmenter fix a perfectly confident wrong-object box?

Reference answer: It can produce a precise mask of the wrong object; target selection remains a separate failure.

06 · CAN A VLM GENERATE SPACE?

CAN A VLM GENERATE SPACE?

06

What if the answer is a spatial sequence?

Fixed park illustration with staged reference annotation

What if the answer is a spatial sequence?

ES 667 · Nipun Batra · authored teaching diagrams28 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

06 · CAN A VLM GENERATE SPACE?

A familiar VLM, a different answer grammar

Visual context and instruction condition a decoder that emits coordinate tokensVision encoderVisual tokensConnector“Where is the red ball?”Language decoder<box><x_120><y_250><x_710><y_830></box>Illustrative coordinate grammar and toy box · not a measured answer

Dedicated head: box coordinates. Sequence decoder: spatial tokens.

ES 667 · Nipun Batra · authored teaching diagrams29 / 56

Build visual encoder and tokens; connector and instruction; decoder; first coordinate token; remaining spatial sequence. This is a generic VLM pattern, not a claim about a specific model’s exact token syntax. The displayed numeric box is a separate toy box.

Ask: Which part makes the output spatial?

Reference answer: Training the sequence model to emit an agreed spatial grammar and coordinates.

06 · CAN A VLM GENERATE SPACE?

Continuous coordinate → bin → token

Architecture diagramnormalized box (0.12, 0.25, 0.71, 0.83)
x1=.12⟶⌊1000(.12)⌋=120⟶<x_120>x_1=.12\quad\longrightarrow\quad \lfloor1000(.12)\rfloor=120\quad\longrightarrow\quad\text{<x\_120>}
y1=.25⟶⌊1000(.25)⌋=250⟶<y_250>y_1=.25\quad\longrightarrow\quad \lfloor1000(.25)\rfloor=250\quad\longrightarrow\quad\text{<y\_250>}
x2=.71⟶⌊1000(.71)⌋=710⟶<x_710>x_2=.71\quad\longrightarrow\quad \lfloor1000(.71)\rfloor=710\quad\longrightarrow\quad\text{<x\_710>}
y2=.83⟶⌊1000(.83)⌋=830⟶<y_830>y_2=.83\quad\longrightarrow\quad \lfloor1000(.83)\rfloor=830\quad\longrightarrow\quad\text{<y\_830>}

Toy convention: 1,000 bins/axis, floor then clip to 999. Decode at bin centres.

Each coordinate becomes a discrete symbol under a declared convention.

ES 667 · Nipun Batra · authored teaching diagrams30 / 56

Reveal x1, y1, x2, y2 separately. This box demonstrates quantization and is not claimed to enclose the dog or ball. Centre decoding makes the maximum normalized error 0.0005 per coordinate, including clipped endpoints. Real models may use different tokenization.

Ask: Do four discrete indices retain infinite spatial precision?

Reference answer: No. Quantization imposes finite coordinate resolution.

06 · CAN A VLM GENERATE SPACE?

Predict one spatial token at a time

Architecture diagramImage + instructionPrefix so farNext-token distribution
<box>
<x_120>
<y_250>
<x_710>
<y_830>
</box>
p(y∣I,q)=∏tp(yt∣y<t,I,q)p(y\mid I,q)=\prod_t p(y_t\mid y_{<t},I,q)

A valid sequence must also describe the correct region.

ES 667 · Nipun Batra · authored teaching diagrams31 / 56

Do not reveal a complete token sequence first. Each advance adds one token after the initial context pipeline. The toy coordinate grammar is not the literal Kosmos-2 vocabulary.

Ask: Could all tokens be syntactically valid but the box be wrong?

Reference answer: Yes. Syntax and spatial correctness are independent checks.

06 · CAN A VLM GENERATE SPACE?

Example: Kosmos-2 grounds generated phrases

Architecture diagramPhrase in generated textTwo corner location tokensBox in a 32 × 32 grid1,024 location tokens · each token represents a grid cell

Spatial grammars are model-specific: Kosmos-2 uses two corner tokens.

Attach name after the generic mechanism. Kosmos-2 links phrase text with location tokens; two tokens specify top-left and bottom-right grid cells. Contrast with the preceding four axis-token toy scheme.

Ask: Why should we not copy the toy x/y token strings into its tokenizer?

Reference answer: Its location vocabulary and serialization convention differ.

06 · CAN A VLM GENERATE SPACE?

Example: Florence-2 changes tasks through prompts

Architecture diagramImage → vision encoderTask promptMultimodal encoder–decoderTask-specific sequence<CAPTION>caption<OD>boxes<CAPTION_TO_PHRASE_GROUNDING>phrases + boxes<REFERRING_EXPRESSION_SEGMENTATION>polygons

The prompt selects an output grammar; a parser turns spatial tokens into geometry.

The displayed task strings are actual processor keys, not the shorthand names from the brief. The referring-expression segmentation route produces polygons; rasterization gives a bitmap. This is an encoder–decoder sequence model.

Ask: Where is the mask bitmap in the segmentation sequence?

Reference answer: The sequence represents polygons; downstream parsing and rasterization produce a mask.

07 · CAN A VLM REASON INTO A MASK?

CAN A VLM REASON INTO A MASK?

07

The target may be implicit in the instruction.

Fixed park illustration with staged reference annotation

The target may be implicit in the instruction.

ES 667 · Nipun Batra · authored teaching diagrams34 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

07 · CAN A VLM REASON INTO A MASK?

From naming to interpreting the target

Fixed park illustration with staged reference annotation

“segment the dog”

“segment the dog beside the red ball”

“segment the object the child is playing with”

Reference target: red ball.
Visible cue: the child’s foot meets the ball.

The instruction can describe a role without naming the object.

ES 667 · Nipun Batra · authored teaching diagrams35 / 56

The park scene stays fixed. The child’s foot visibly contacts the ball. Even with a clear authored cue, action interpretation is an inference, and real images may be ambiguous. These reference answers do not claim measured model reasoning.

Ask: Which noun in the final request directly says “ball”?

Reference answer: None. The model must resolve the described interaction.

07 · CAN A VLM REASON INTO A MASK?

The request can depend on world knowledge

Authored rain scene with umbrella, ball and bag; world-knowledge query targets umbrella

“segment the object used to stay dry in rain”

Umbrella

A second controlled illustration.
No umbrella is hidden in the park scene.

Interpret the request, then identify the pixels of the chosen object.

ES 667 · Nipun Batra · authored teaching diagrams36 / 56

This supplementary scene makes the world-knowledge query answerable without inventing an absent park object. Do not display hidden chain-of-thought; just the instruction and target.

Ask: Would an excellent mask of the ball answer this instruction?

Reference answer: No. Correct shape is insufficient when target identity is wrong.

07 · CAN A VLM REASON INTO A MASK?

Turn a language representation into a mask

VLM segmentation hidden state and spatial image features condition a mask decoderImage → VLM visionInstructionVLMinterprets the request<SEG>token hidden state
hSEGh_{\mathrm{SEG}}
ProjectionImage pixel encodersame image, spatial detailMask decoderpixel mask

Language interprets WHAT is meant; the mask decoder predicts WHICH PIXELS.

ES 667 · Nipun Batra · authored teaching diagrams37 / 56

First image and query enter a VLM. Then introduce SEG; expose its hidden representation; project it into a segmentation prompt; show spatial image features entering the mask decoder; finally emit a mask. The displayed hidden state is a symbol, not decoded thought.

Ask: Why must the mask decoder also receive image features?

Reference answer: The prompt specifies the target; spatial image features supply location and boundary detail.

07 · CAN A VLM REASON INTO A MASK?

Example: LISA-style reasoning segmentation

VLM segmentation hidden state and spatial image features condition a mask decoderImage → VLM visionInstructionVLMinterprets the request<SEG>token hidden state
hSEGh_{\mathrm{SEG}}
ProjectionImage pixel encodersame image, spatial detailMask decoderpixel mask

A learned segmentation representation connects a VLM to a pixel decoder.

Name LISA after the mechanism. The original implementation projects the selected token hidden representation into a SAM-style prompt and combines it with image embeddings. No full paper architecture or private reasoning transcript is needed.

Ask: How does this differ from text → box → SAM?

Reference answer: It supplies a learned segmentation prompt from the VLM rather than requiring an intermediate box.

08 · FOUR ARCHITECTURES FOR LANGUAGE-GROUNDED VISION

FOUR ARCHITECTURES FOR LANGUAGE-GROUNDED VISION

08

Compare where language enters, what leaves, and what is trained.

Fixed park illustration with staged reference annotation

Compare where language enters, what leaves, and what is trained.

ES 667 · Nipun Batra · authored teaching diagrams39 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

08 · FOUR ARCHITECTURES FOR LANGUAGE-GROUNDED VISION

Four ways to make language spatial

Architecture diagramOpen vocabularytext vectors → matchingboxes + scoresalignment + boxPhrase groundingtext ↔ features / queriesreferred boxesalignment + boxGround + segmenttext → box → SAMpixel masksgrounder + maskGenerative / reasoninginstruction → VLMtokens OR SEG → masktoken OR LM + mask

Language entry → output representation → learning signal.

ES 667 · Nipun Batra · authored teaching diagrams40 / 56

Reveal one complete family per click. The fourth row includes two distinct output paths: generated coordinates versus a segmentation representation plus pixel decoder. Categories describe patterns, not mutually exclusive taxonomic bins.

Ask: Which design naturally exposes an intermediate box?

Reference answer: The modular ground-and-segment system. A learned SEG interface need not expose one.

08 · FOUR ARCHITECTURES FOR LANGUAGE-GROUNDED VISION

A recorded answer can miss the intended target

Actual recorded Grounding DINO boxes, not reference targets

“ball beside the dog.”

dog · 0.603
ball · 0.476

Recorded alignment scores, not accuracy.
The requested target is the ball.

Keep the raw predictions. Inspect which words and regions they link.

Measured content is loaded from results/grounded_vision_outputs.json. These are actual Grounding DINO tiny outputs on the authored scene at box/text thresholds 0.25. The detector returns noun detections, not a guaranteed single referred instance. Explore all five interfaces, color swap and absent bicycle; use reference mode for unmeasured generative/reasoning examples.

Ask: Which returned box answers the request?

Reference answer: The ball box; the dog is the reference object. Returning both does not resolve the referring expression.

09 · TRAINING / EVALUATION SYNTHESIS

TRAINING / EVALUATION SYNTHESIS

09

The requested output determines supervision and evaluation.

Fixed park illustration with staged reference annotation

The requested output determines supervision and evaluation.

ES 667 · Nipun Batra · authored teaching diagrams42 / 56

Use this transition to restate the problem before introducing a mechanism.

Ask:

Reference answer:

09 · TRAINING / EVALUATION SYNTHESIS

What annotation teaches each behavior?

Architecture diagramimage + classglobal semanticsimage + box + classwhere + whatimage + phrase + boxwhich regionimage + phrase + maskwhich pixelsimage + instruction + spatial sequencestructured answerimage + reasoning query + maskimplicit target

Richer instructions require targets that make their intended region identifiable.

ES 667 · Nipun Batra · authored teaching diagrams43 / 56

Three training slides total. The ladder represents different annotation types, not an automatic performance ranking. Mask supervision can be paired with explicit or implicit descriptions.

Ask: Can image-level class labels alone specify an exact mask?

Reference answer: No. They do not directly supply spatial boundary targets.

09 · TRAINING / EVALUATION SYNTHESIS

Loss follows the output representation

Box detector
Lalign+λbLbox\mathcal L_{\mathrm{align}}+\lambda_b\mathcal L_{\mathrm{box}}
Mask decoder
Lmask\mathcal L_{\mathrm{mask}}
Generated coordinates
LLM=−∑tlog⁡p(yt∣y<t,I,q)\mathcal L_{\mathrm{LM}}=-\sum_t\log p(y_t\mid y_{<t},I,q)
VLM + mask
LLM+λmLmask\mathcal L_{\mathrm{LM}}+\lambda_m\mathcal L_{\mathrm{mask}}

Train the semantic choice and the spatial answer the system actually produces.

Box objectives commonly combine alignment/classification and geometry such as L1/GIoU after assignment. Mask objectives compare pixels or overlap. Original SAM uses focal and Dice plus quality prediction; LISA uses LM with BCE/Dice. No full derivation in main route.

Ask: Does a coordinate token loss directly optimize IoU?

Reference answer: No. It penalizes next-token errors; spatial quality must still be evaluated.

09 · TRAINING / EVALUATION SYNTHESIS

A teacher can scale labels—and its mistakes

Architecture diagramUnlabelled image + textTeacher boxes / masksStudent supervision

A wrong instance can become a very precise training target.

Audit relations, absence and domain shift. Keep a human-checked held-out set.

More pseudo-labels do not automatically mean better grounding.

Third and final training slide. Link to GLIP-style data expansion without a long training recipe. Teacher confidence cannot replace instance-level validity checks.

Ask: What error can survive perfect mask boundaries?

Reference answer: A wrong region referred to by the instruction.

09 · TRAINING / EVALUATION SYNTHESIS

Evaluate the output and the target identity

Architecture diagramBOXIoU + detection APMASKIoU / DiceREFERRING PHRASEcorrect instance + localizationGENERATED ANSWERvalid syntax + spatial correctness

State the matching rule, threshold and split; one metric cannot test every failure.

First of two evaluation slides. AP aggregates ranked detections under a defined matching protocol. Referring expression accuracy often uses a box-IoU threshold but also requires the intended instance. Mask IoU/Dice depend on aggregation choice. Arithmetic is in backup.

Ask: Why is valid token syntax insufficient?

Reference answer: The generated box can refer to the wrong object or be shifted.

09 · TRAINING / EVALUATION SYNTHESIS

Separate four failure modes

Architecture diagramright shapewrong instanceright objectshifted coordinatesright boxbad maskabsent bicyclehallucinated

Diagnose target choice, coordinate mapping, shape and absence separately.

ES 667 · Nipun Batra · authored teaching diagrams47 / 56

Second and final evaluation slide. The small drawings intentionally illustrate errors, not real predictions. Shifted coordinates show both reference and displaced box; shape failure erases part of the dog mask.

Ask: Which component should we inspect for a consistent shift?

Reference answer: The coordinate transforms and box format before blaming language understanding.

09 · TRAINING / EVALUATION SYNTHESIS

Choose a route; explain the representation

Fixed park illustration with staged reference annotation

“segment the object the child is playing with”

Where does language enter?
What spatial representation leaves?
What target and loss teach it?

Selection → representation → pixels

Language changes the target. Architecture determines how that choice becomes space.

ES 667 · Nipun Batra · authored teaching diagrams48 / 56

Close by asking students to propose either modular grounding+SAM or VLM+mask, then identify the loss and a failure case. Do not imply one design always wins. Main lecture ends here.

Ask: How would you tell whether a correct mask came from understanding the instruction?

Reference answer: Use counterfactual queries and same-class alternatives, inspect target identity and score spatial quality separately.

OPTIONAL / BACKUP

Coordinate handoff: the same box in two systems

Architecture diagramDetector coordinatesUndo resize / padOriginal-image boxApply SAM transform
(x,y)orig=((xdet−px)/sd, (ydet−py)/sd)(x,y)_{\mathrm{orig}}=((x_{\mathrm{det}}-p_x)/s_d,\ (y_{\mathrm{det}}-p_y)/s_d)

SAM mask postprocessing: undo SAM resize/padding → original-image mask.

Box format and coordinate frame must be explicit at the interface.

Example: original 800×500; detector resize s=.8 then pad top=120 to 640×640. Detector xyxy [96,296,272,440] → original [120,220,340,400]. SAM longest-side scale1.28 → [153.6,281.6,435.2,512]. Original SAM pads bottom/right, so no added top/left offset. Use library postprocessors.

Ask:

Reference answer:

OPTIONAL / BACKUP

Overlap arithmetic belongs in backup

Worked overlapping boxes: IoU 0.60 and Dice 0.75Two 200 × 200 boxesIntersection = 150 × 200 = 30,000Union = 40,000 + 40,000 − 30,000IoU = 30,000 / 50,000 = 0.60Dice = 60,000 / 80,000 = 0.75
IoU=∣A∩B∣∣A∪B∣=3000050000=0.60,Dice=2∣A∩B∣∣A∣+∣B∣=0.75\mathrm{IoU}=\frac{|A\cap B|}{|A\cup B|}=\frac{30000}{50000}=0.60,\quad\mathrm{Dice}=\frac{2|A\cap B|}{|A|+|B|}=0.75

For nonempty binary masks, Dice = 2 IoU / (1 + IoU).

ES 667 · Nipun Batra · authored teaching diagrams50 / 56

Two 200×200 boxes offset by50 pixels: intersection30000, union50000. This binary evaluation identity is not a complete specification of a differentiable Dice training loss. Define empty-mask behavior and macro/micro aggregation.

Ask:

Reference answer:

OPTIONAL / BACKUP

Pseudo-labels need an error audit

Architecture diagramTeacher predicts regionFilter / inspect labelsStudent learns targets

Keep teacher version, threshold, prompt, image transform and human-audit split.

Confidence filtering cannot repair systematic target confusion.

Use a stratified audit: category, relation, small objects, absent query, new domain. Prevent train/evaluation leakage. Retain low-confidence cases for diagnosis instead of hiding every failure.

Ask:

Reference answer:

OPTIONAL / BACKUP

Optional: language can point into masks

Fixed park illustration with staged reference annotation

“The [dog] is beside the [child].”

[dog] → dog mask
[child] → child mask

GLaMM connects grounded language and region masks within conversation.

One optional contextual slide. The sentence and masks are illustrative, not a measured output. No additional architecture is required.

Ask:

Reference answer:

OPTIONAL / BACKUP

Specify the evaluation protocol

Architecture diagramIntended referentPrediction matchingGeometry / pixels

Box AP: thresholds + class matching + ranked detections.
Referring accuracy: target identity + chosen IoU threshold.
Mask overlap: per-object / per-image / global aggregation.
Absence: false positives on negative queries.

A score without its protocol is not a comparable result.

Retained evaluation detail outside the main flow. Avoid comparing headline AP, mask mIoU and referring accuracy as if they measured the same task.

Ask:

Reference answer:

OPTIONAL / BACKUP

Optional: carry a spatial prompt through video

Architecture diagramPrompted frameImage features + memoryMasks across frames

SAM 2 extends promptable segmentation to video.
Language selection still depends on the surrounding system.

Temporal propagation adds a memory problem beyond this image-only lecture.

Preserves the prior optional extension. This is not needed for the 55-minute route.

Ask:

Reference answer:

OPTIONAL / BACKUP

Primary sources and implementation references

CLIP · 2021

OWL-ViT · 2022

GLIP · 2022

DETR · 2020

Grounding DINO · 2024

Segment Anything · 2023

Grounded SAM · implementation

Kosmos-2 · 2023

Original diagrams; illustrative examples are labelled separately from recorded inference.

ES 667 · Nipun Batra · authored teaching diagrams55 / 56

Primary sources checked 4 October 2026. The lecture teaches representative published designs, not a leaderboard.

Ask:

Reference answer:

OPTIONAL / BACKUP

Primary sources and implementation references

Florence-2 · 2024

Florence-2 · task parser

LISA · 2024

LISA · segmentation interface

GLaMM · 2024

COCO · evaluator

SAM 2 · 2024

Original diagrams; illustrative examples are labelled separately from recorded inference.

ES 667 · Nipun Batra · authored teaching diagrams56 / 56

Primary sources checked 4 October 2026. The lecture teaches representative published designs, not a leaderboard.

Ask:

Reference answer: