Interactive Explainer
Image Segmentation, with a Real Segmenter
Classification labels the image. Detection labels boxes. Segmentation labels every pixel. This page runs a real pretrained DeepLab v3+ segmenter on whatever photo you hand it, then lets you paint your own mask and watch mean-IoU and pixel accuracy tick up toward the model's output.
A real segmenter running in your browser
The model below is DeepLab v3+ (MobileNetV2 backbone) trained on the Pascal VOC benchmark's 21 classes (background + 20 foreground: person, cat, dog, bicycle, car, chair, sofa, tv, …). TensorFlow.js downloads it once (~10 MB) and runs every inference locally on your GPU.
Three flavours of segmentation to keep in mind:
- Semantic: per-pixel class. Two people merge into one "person" blob.
- Instance: per-pixel class and instance ID. Two people get two masks.
- Panoptic: the union—"things" (countable) get instance IDs, "stuff" (sky, road) just gets a class. DeepLab does semantic; Mask R-CNN / Mask2Former handle the others.
Pick a photo Loading segmenter…
Six CC-licensed stock photos with Pascal VOC classes.
Classes detected in this image
A segmentation is a function from pixels to labels
The model's output is a $W \times H \times C$ tensor of class logits—one logit per pixel per class. Taking the argmax at every pixel collapses it to a label map with one integer per pixel. Below is the model's raw label map rendered as a colour overlay. Every pixel has exactly one colour, because every pixel has exactly one winner.
Move the mouse over the image to see what class the model assigned to that exact pixel.
Paint your own mask; grade it live
The canvas on the left is the image (the model's mask faintly overlaid as a guide). The canvas on the right is your blank mask. Pick a class, paint, and watch mean-IoU, pixel accuracy, and per-class IoU update after every stroke—against the real model output as ground truth.
Per-class IoU
Region growing: the pre-neural baseline
Before CNNs, segmentation started from a seed pixel and grew outward while neighbours stayed similar in colour. The rule is four lines of code: BFS over 4-connected neighbours; include a pixel if its RGB distance to the seed (or the running mean of the region) is within a threshold $\tau$.
Compare its output to the neural network's on the same photo.
Architecture families that beat region growing
| Family | Key move | Representatives | Strength |
|---|---|---|---|
| Encoder-decoder | CNN downsamples to coarse features, then upsamples; skip connections glue fine detail back. | FCN, U-Net, SegNet | Clean medical-imaging masks; small, fast, interpretable. |
| Dilated / multi-scale (this page) | Keep high resolution; grow receptive field with dilated conv + atrous spatial pyramid pooling. | DeepLab v1–v3+, PSPNet | Big receptive field without resolution loss; great on natural images. |
| Mask-prediction / Transformer | Decoder queries emit (class, binary mask) pairs directly—no per-pixel argmax. | Mask R-CNN, MaskFormer, Mask2Former, SAM | Natively handles instance and panoptic; state of the art. |
Why cross-entropy alone fails
Per-pixel cross-entropy is the obvious loss, but on most real images 80% of pixels are background. A model that always predicts "background" gets 80% accuracy while being useless. Three losses fix the imbalance:
- Dice loss (Milletari et al., 2016): maximize the soft Dice coefficient. Numerator and denominator shrink together, so the model still gets signal on rare classes.
- IoU / Lovász: directly optimize the metric you care about. Lovász makes it differentiable.
- Focal (Lin et al., 2017): cross-entropy re-weighted so easy pixels contribute less. Mostly a detection trick, but widely borrowed for segmentation too.
The big number
A segmentation prediction is one decision per pixel. Scale matters: even a small photo has hundreds of thousands of them.
Per-pixel predictions for this photo
Every one has to get a class. A 99% accurate model still gets thousands of pixels wrong—which usually shows up as jagged boundaries and missed thin structures.
Four things that still trip people up
"Pixel accuracy tells you if the model is good."
On a sky-heavy photo, the background-predicting trivial model
wins. Report mean IoU alongside pixel accuracy, always.
"A mask that's a superset is safe."
A mask that covers the truth plus extra has IoU =
truth / mask, which shrinks as the mask grows. Tight wins;
sprawl is penalised.
"Segmentation is classification, done more."
Per-pixel argmax would give flickery, isolated wrong pixels.
Real architectures inject structural priors: skip
connections, dilated context, CRFs, decoder queries.
"IoU 0.9 looks perfect."
It can still miss a 10-pixel telephone wire threading across
the image—maybe the one pixel your autopilot needs.
Qualitative failure review is not optional.
The three flavours of segmentation
- Semantic segmentation. Each pixel gets a class. Two cars next to each other are one "car" blob. The standard task this article focuses on. Datasets: ADE20K, Cityscapes (19 classes), PASCAL VOC.
- Instance segmentation. Each pixel gets a class and an instance ID. The two cars become car-1 and car-2. Mask R-CNN is the classical approach: detect bounding boxes, segment within each. Datasets: COCO, LVIS.
- Panoptic segmentation. Combines both — "things" (countable: cars, people) get instance IDs; "stuff" (uncountable: sky, road) gets a class only. The COCO-Panoptic and Cityscapes-Panoptic datasets unified the field around this around 2018.
The 2021+ trend has been to unify all three under a single transformer-decoder architecture: Mask2Former, Mask DINO, OneFormer. One model, one set of weights, all three tasks.
SAM — the foundation model for segmentation
Segment Anything (Kirillov et al., 2023; SAM-2, 2024) is the segmentation analogue of GPT-3 for text: pretrain a huge model on a billion masks, then have it segment anything from a prompt (point, box, or text). Architecture: ViT image encoder + small prompt encoder + lightweight mask decoder. Output: zero-shot, high-quality masks at interactive latency.
- Strengths. Zero-shot on novel objects; interactive (click a point and get a mask); production-ready in 2024.
- Weaknesses. No class labels (SAM segments, doesn't classify); fine-grained boundaries on small objects still imperfect; needs a separate classifier on top for semantic labels.
- SAM-2 (2024). Adds video support — propagate masks across frames with memory.
The right mental model: SAM is the high-recall "find every object" tool; you compose it with a classifier (CLIP, Grounding-DINO) for "find every object of class X" pipelines.
Loss functions for segmentation
- Pixel-wise cross-entropy. The default. Fast, well-understood, works well when classes are roughly balanced.
- Weighted cross-entropy. Reweight rare classes. Important for medical segmentation where the lesion might be 1% of pixels.
- Dice loss. $1 - 2|A \cap B| / (|A| + |B|)$. Directly optimises overlap; very effective for imbalanced classes. Most medical segmentation papers use Dice or Dice + CE.
- Focal loss. Down-weights easy examples ($(1-p_t)^\gamma$ factor). Helps when many easy negatives dominate gradient.
- Boundary loss. Penalises errors near class boundaries more — encourages sharp, accurate edges.
- Cross-entropy + Dice combo. Most practical recipes blend the two. CE for stable gradients, Dice for overlap optimisation.
Reading list
- Long, Shelhamer, Darrell (2015) — Fully Convolutional Networks for Semantic Segmentation. The paper that defined the field.
- Ronneberger, Fischer, Brox (2015) — U-Net. See the dedicated U-Net article.
- Chen et al. (2017) — DeepLab. Atrous / dilated convolutions for high-resolution features.
- He et al. (2017) — Mask R-CNN. Instance segmentation gold standard.
- Cheng, Schwing, Kirillov (2022) — Mask2Former. The unifying transformer architecture.
- Kirillov et al. (2023) — Segment Anything.
- Ravi et al. (2024) — SAM 2. Video extension.