Learning outcomes

By the end of this lecture you will be able to:

1. Extend a classifier to localization + detection with multiple heads.
2. Compute IoU and explain NMS.
3. Contrast R-CNN family vs YOLO vs DETR.
4. Pick anchor boxes and explain delta parameterization.
5. Use a U-Net for per-pixel segmentation; choose appropriate loss.
6. Prompt SAM for zero-shot segmentation.

Recap · where we are

Module 4 so far:

• L7 · CNN mechanics, receptive field, classic architectures
• L8 · Modern CNNs (ResNet bottleneck, MobileNet, EfficientNet) + transfer learning

All of that was classification — one label per image.

Today's jump: one label per pixel, per object, per region.

Four questions

1. How do we go from classification to bounding boxes?
2. How do we compare boxes — what is IoU? How do we deduplicate predictions — NMS?
3. How does YOLO do all of it in one forward pass?
4. How do we predict a mask per pixel — U-Net and beyond?

The spectrum of "what's in this image"

Today we cover the last four.

Training for two goals at once

Classification + localization · the multi-task loss

Keep the CNN. Add two heads:

class ClsLocHead(nn.Module):
    def __init__(self, feat_dim, n_classes):
        super().__init__()
        self.cls = nn.Linear(feat_dim, n_classes)   # class scores
        self.box = nn.Linear(feat_dim, 4)           # (x, y, w, h)

For one image:

• True class , model predicts class logits.
• True box , model predicts .

is the balancing knob: → box matters more; → class matters more. Typical is Smooth L1 (L2 near 0, L1 far → robust).

Worked numeric · multi-task loss

One image of a cat. True class index 1. True box .

Model predicts:

• Class logits → confidently "cat" → say .
• Box .

Box loss (MSE).

Total ().

This 3.18 is what backprop sees → updates both the class head and the box head simultaneously.

IoU · the metric · NMS · the cleanup

IoU · visual examples

IoU · a quick worked example

Predicted box · · area = 100 × 100 = 10,000.
Ground-truth box · · area = 10,000.

Intersection · · area = 50 × 50 = 2,500.
Union · 10,000 + 10,000 − 2,500 = 17,500.

NMS · the "shouting answers" analogy

Concretely · sort by confidence, keep the best, discard any later box that overlaps it (IoU above threshold). Repeat until none left. Standard since R-CNN; survives in YOLO and most detectors.

NMS · step-by-step example

5 predicted boxes for one car. IoU threshold 0.5.

Step 1 · pick A (highest score). Add to keep.
Compare IoU(A, others): IoU(A,B)=0.8 → drop B; IoU(A,C)=0.2 → keep; IoU(A,D)=0.7 → drop D; IoU(A,E)=0.1 → keep.
Pool now [C, E].

Step 2 · pick C. Add to keep.
IoU(C, E) = 0.15 → keep E.
Pool now [E].

Step 3 · pick E. Add to keep. Pool empty. Stop.

Final · keep = [A, C, E]. 5 boxes → 3 final detections.

NMS · pseudocode

def nms(boxes, scores, iou_threshold=0.5):
    idx = scores.argsort(descending=True)
    keep = []
    while len(idx):
        i = idx[0]
        keep.append(i)
        # drop any later box that overlaps >threshold with this one
        idx = [j for j in idx[1:] if iou(boxes[i], boxes[j]) < iou_threshold]
    return keep

Greedy — highest-confidence box wins in its neighborhood. Every detector uses some form of NMS (Faster R-CNN, YOLO) or its learned replacement (DETR's Hungarian matching).

How do we grade a detector?

Computing AP · a worked example

3 ground-truth cats. 5 predictions, sorted by confidence. IoU > 0.5 → True Positive.

AP = area under the precision–recall curve built from these points.
mAP = average AP over all classes.

mAP@0.5 = AP at IoU threshold 0.5. mAP@[0.5:0.95] = average over 10 IoU thresholds (COCO standard).

R-CNN family · two-stage (brief)

R-CNN (2014) → Fast R-CNN (2015) → Faster R-CNN (2015)

Two-stage idea:

1. Propose candidate regions (where might objects be?).
2. Classify each region (what's in it?).

Evolution:

YOLO · you only look once

YOLO · the grid-of-responsibilities idea

Divide the image into a 7×7 grid. Each cell fills out a form with three questions:

1. Is there an object centred here? (yes/no + confidence) → objectness loss ()
2. If yes, where exactly? (4 numbers ) → box loss ()
3. If yes, what class? (cat / dog / car…) → class loss ()

YOLO's total loss = sum of all three across every cell:

• Box active only if a cell contains an object. MSE on (sqrt so small boxes matter).
• Objectness active always — BCE pushes toward 1 if object, 0 if not.
• Class active only if cell contains an object — cross-entropy.

Worked numeric · YOLO loss for one cell

Cell holds a dog (class index 2).

Ground truth. Box . Objectness 1. Class one-hot .

Prediction. . Objectness 0.85. Probs .

Components.

Total per cell (no weighting): . Sum across all cells = full image loss.

Why predict deltas · relative directions

Decoding the box prediction · with example

Anchor is known per cell. Network predicts deltas .

Centre — small shift, scaled by anchor size:

Size — log-space correction; exp keeps width/height positive:

(If , → width unchanged.)

Worked numeric. Anchor , deltas .

Final box · . Network only had to learn small corrections, anchor did the heavy lifting.

Speed vs accuracy · a detector comparison

The YOLO lineage

DETR · ditch the post-processing

YOLO and Faster R-CNN are conceptually two-step: predict densely (thousands of boxes), then clean up with NMS.

DETR's question · can we just directly predict the final, clean set?

It outputs a fixed-size set of 100 predictions. No grid, no anchors, no NMS.

Challenge · the model outputs 100 boxes; the ground truth might have only 3. Prediction order is arbitrary — pred #47 might match ground truth #1.

DETR cleans up detection conceptually but is slower and data-hungry. YOLO still wins speed; DETR wins elegance.

From detection to segmentation

Detection gives boxes around objects. Segmentation gives a label per pixel.

Key architectural change: we need to go back up in spatial resolution — the feature map shrinks through convs/pooling, but the output must match the input size.

Solution: encoder-decoder with upsampling.

U-Net architecture

U-Net · with channels and sizes

U-Net skips · the cheat sheet analogy

Result · the decoder uses high-level summary for what and the cheat sheet for exactly where the boundary is. Net effect · sharp accurate masks.

Why skip connections are essential

The encoder compresses spatial info into richer features. But spatial precision is lost — a 16×16 feature map can't localize edges accurately.

Skip connections let the decoder concatenate encoder features at matching resolution:

def forward(self, x):
    e1 = self.enc1(x)         # 128×128
    e2 = self.enc2(pool(e1))  #  64×64
    e3 = self.enc3(pool(e2))  #  32×32
    b  = self.bridge(pool(e3))

    d3 = self.dec3(cat([up(b),  e3]))    # ← skip from e3
    d2 = self.dec2(cat([up(d3), e2]))    # ← skip from e2
    d1 = self.dec1(cat([up(d2), e1]))    # ← skip from e1
    return self.final(d1)

Every modern segmentation net (DeepLab, SegFormer) uses this pattern.

Segmentation loss · IoU to Dice

For boxes we used IoU. For masks we can do the same:

Good metric — but its gradient is "sharp," tricky for SGD. The Dice coefficient is the smoother cousin:

Optimizers minimize, so:

Perfect prediction → Dice 1 → loss 0. No overlap → loss 1.

Why it handles imbalance. It only counts pixels in or — the vast background of true negatives doesn't dominate (unlike plain CE).

Worked numeric · Dice loss on a 2×2 image

Task · segment the top-left pixel.

Ground truth , prediction (probabilities) .

1. Intersection

The model backprops this 0.28.

Other segmentation losses

• Pixel-wise cross-entropy · default. Per-pixel softmax over classes.
• Focal loss · down-weights easy pixels → model focuses on boundaries.
• Boundary loss · explicit pixel accuracy near edges.

Why U-Net caught on in medical imaging

Ronneberger 2015 targeted electron microscopy cell segmentation. Two reasons it spread:

1. Small data tolerance · medical datasets are in the hundreds. U-Net's strong spatial prior (encoder-decoder + skip) makes this workable.
2. Permissive license + simple architecture · anyone could port it to any framework in a day.

By 2020, U-Net was the default segmentation network not just in medicine but in satellite imagery, materials science, audio-spectrogram analysis, and later diffusion models (L21 / L22) — where the same encoder-decoder-with-skips handles noise-to-image mapping.

Instance segmentation · Mask R-CNN (brief)

Built on Faster R-CNN. Adds a third head:

• Class head (from Faster R-CNN)
• Bbox head (from Faster R-CNN)
• Mask head — a small FCN producing a pixel mask per region

Segment Anything · SAM (Meta, 2023)

A foundation model for segmentation:

• Trained on 1.1 billion masks across 11M images.
• Takes a prompt — a point, a box, or text — and returns a mask.
• Zero-shot on unseen categories and domains.

The fixed-dictionary problem

The open-vocabulary shift · how it works

Embedding · a vector that represents data. Image embedding = vector representing an image; text embedding = vector representing text.

CLIP · OpenAI 2021. Trained on millions of (image, caption) pairs. Learns a shared embedding space: the vector for a dog photo lands close to the vector for "a photo of a dog"; both far from "a photo of a cat."

How open-vocab detectors use this:

1. User provides a text prompt — e.g. "a red bicycle".
2. Model computes the text embedding via CLIP's text encoder.
3. Model processes the image with a CNN → spatial features.
4. Search the image for regions whose embeddings are close to the text embedding.
5. Match → draw a box.

Examples · OWLv2, GroundingDINO, SAM-with-text. The 2024–2026 frontier is fully prompt-driven vision.