Beyond Boxes

Read the question first, then the target. Classification can mean a single scene label or a vector of category-presence labels. The street contains several categories, so a single object-class softmax would not list everything present. Detection supplies a variable number of records. The examples illustrate task definitions rather than a benchmark annotation policy. Begin with the image alone. Reveal one question and its visual target at a time. The persistent family labels distinguish global outputs, sets, dense fields, and query-conditioned outputs; these are useful organizational views rather than mutually exclusive model classes.

Reveal the mask after discussing the box. The teal outline is a manually authored, coarse silhouette. It is neither official ground truth nor a model prediction. Visible-region segmentation excludes occluded parts unless the task explicitly asks for an amodal mask.

Use the road and sky first, then the people. Only selected regions are coloured so the source image remains legible. This is not an exhaustive semantic annotation: buildings, vegetation, bicycle, and a distant pedestrian are deliberately left unpainted. Colours encode class identity in this illustration.

Start with the selected pixel on the foreground person in the street. Reveal its target class, then the feature vector at that location, then C logits. The magnified pixel patch is an authored semantic illustration around that same location; it is not a dataset annotation. The classifier sees contextual features, not only one RGB triplet.

The same learned classifier operates at every spatial location. Softmax normalizes over C classes at each pixel separately. The illustrative logits log(0.10), log(0.70), and log(0.20) give probabilities exactly 0.10, 0.70, and 0.20. This keeps one numerical example throughout the pixel trace.

Hold the target person and probabilities 0.10, 0.70, 0.20 fixed. Reveal the negative log calculation, then draw its 0.357 value back onto the selected location in the schematic loss grid. Other cells remain blank until the image-loss slide. The calculation is illustrative and uses natural logarithms.

Show one loss per valid pixel. The highlighted pixel retains the same 0.357 value from the previous slide. A grey crossed cell is ignored. Reveal the mean over V only after the students see the spatial loss field. Raw logits and integer targets are the usual cross-entropy inputs; argmax is not part of training.

The encoder computes contextual features on a smaller grid. A spatial decoder increases resolution and combines features. A final class head maps each location to C logits. This is a schematic, not a particular implementation: outputs may first be produced at a lower resolution and then resized to the target grid.

This is an explicit toy compression, not a claim that a CNN uses average pooling at every stage. The 8 × 8 binary mask is averaged within four 4 × 4 blocks. The coarse values are 0.5, 0, 0.5, and 0.125. Nearest-neighbour upsampling repeats them. A learned decoder can exploit training priors, but it cannot guarantee recovery of arbitrary detail discarded from its inputs.

First trace the deep path through upsampling. Then reveal the skip from the fine encoder stage. Match the spatial dimensions before fusion. U-Net concatenates the encoder and decoder feature channels and applies convolutions. This is a feature concatenation, unlike an additive residual connection. The network still has to learn how to use the features. Do not name the connection on its first build. First ask where detail could come from, then reveal the fine feature branch, then name the skip and the fusion operation.

Reveal the skip paths after following the U-shaped computation. This drawing omits channel counts, convolution blocks, and the final class head for clarity. Original U-Net used valid convolutions and cropped skip tensors before concatenation; modern variants often use padding. The conceptual claim is aligned feature fusion, not exact original tensor dimensions.

Long et al. showed how a classification network could produce dense predictions using convolutional score maps and learned upsampling. The skip variants combine coarse semantic scores with finer layers. The drawing summarizes score fusion and omits the final upsampling step. This is a short connection, not an architecture survey.

The probabilities can be retained for confidence analysis, but the hard semantic prediction is an integer map. Argmax of logits gives the same winning class as argmax of softmax probabilities. Training uses differentiable scores; hard labels are convenient for display and discrete overlap metrics.

Use the exact same 4 × 4 masks throughout the builds. Ground truth has 6 foreground pixels; prediction has 8. First ask students to count overlap, then reveal errors, then show the quotient. Background agreement is outside the foreground union. These are synthetic labels and predictions, not measured model outputs. Use green for TP, coral for FP, and amber for FN. Build 3 shows counts; build 4 calculates the quotient.

The miniature 10 × 10 image has exactly six foreground pixels. Explain why accuracy is dominated by the 94 background pixels. Mean IoU averages class-wise overlap, subject to the benchmark’s absent-class convention. Report the class of interest and define which pixels and classes participate.

The selected foreground pedestrians and cyclist are all people. The semantic illustration gives them the same colour. The reveal assigns different IDs through different colours. It does not change their shared class. A distant pedestrian is not annotated in this selected-region illustration. Avoid interpreting the coloured sample as an exhaustive instance set.

An instance record can include a class, a box, and an image-sized binary mask. Boxes are useful for the detector-based architecture we will use. Some methods predict masks without a separate box stage. Unlike semantic class channels, the instance dimension N is variable. This reconnects briefly with Part I’s set target without repeating candidate assignment. Show the old class-and-box record first, then append the mask M_i. The records remain an unordered variable-size set.

Begin with the region feature and class/box outputs already familiar from object detection. Ask for the additional output, then reveal a local mask. Only after this motivation show how region features are aligned and how Mask R-CNN implements the branches.

A detector supplies proposed regions over a shared image feature map. RoIAlign samples features using bilinear interpolation without the coordinate rounding of RoIPool. It produces fixed spatial dimensions for different region sizes. The schematic follows one dog proposal. Feature extraction is shared across regions.

Reveal the class branch, the box branch, then the mask branch. The mask branch is a convolutional spatial predictor running in parallel with the class and box heads. It is not obtained by rasterizing the predicted box. The full system also includes the region proposal network and detector losses, already familiar from the object detection lecture.

The displayed 28 × 28 binary mask is rasterized from the same authored dog polygon and box used by the lab; it is not a measured model output. Original Mask R-CNN commonly uses a 28 × 28 mask grid. At inference, select the predicted class mask, resize it to the detected box, and threshold to obtain a binary mask. Align resizing and coordinate conventions with the implementation.

Mask R-CNN separates class prediction from mask prediction. For each positive matched region, the target class chooses the channel for the mask loss. At inference the predicted class chooses it. Background regions do not receive an object mask loss. Loss terms shown are schematic unit-weight sums; the paper also includes proposal training. Keep class-specific masks distinct from one semantic softmax per pixel. Keep the main explanation to the three losses on a matched positive region; class-channel mechanics belong in these notes.

One slide is sufficient. A panoptic output assigns each valid pixel a semantic class and an instance identifier for things, using a single non-overlapping partition. Stuff does not require object instance identity. The selected-region overlay illustrates the distinction; it is not a complete benchmark panoptic map. The things/stuff taxonomy depends on the dataset.

Use the recurring street for context, then compare the two schematic skeletons. Amber circles are joints and blue segments show known anatomical connections. The photo overlay is authored and approximate. Keypoint annotations are image coordinates, not automatically a 3D body model.

DeepPose is a direct coordinate-regression example. The numerical crop calculation is illustrative, not measured from the street image or skeleton. Normalized x is multiplied by crop width and normalized y by crop height, then the crop offset can be added to recover image coordinates. Visibility and validity are related but not identical. A common coordinate loss is squared error over supervised joints.

Targets often place a Gaussian bump around an annotated coordinate; a common objective is squared error between predicted and target heatmaps. The displayed maps are illustrative. The hard peak can be decoded with argmax, with optional subpixel refinement or a soft-argmax method. Heatmap dimensions h × w may be smaller than the image and require coordinate rescaling. In a top-down pipeline, predict K maps per detected person.

The grey map only illustrates near/far ordering. It is not a predicted or physically measured depth map and does not show realistic surface variation. Darker means nearer here; other visualizations reverse the convention. Depth often means camera-axis z distance rather than Euclidean range along a ray. The dataset must define it. Sky may not have a finite valid depth label.

The L1 expression is a simple teaching loss, not a claim that the cited Eigen system uses only L1. Depth methods may operate in log depth or inverse depth and may include scale-invariant, gradient, or geometric objectives. The purpose is to distinguish categorical per-pixel cross-entropy from continuous per-pixel regression.

The ray diagram is a geometric illustration. A monocular model can learn useful scale priors from data, object sizes, and camera information. Those priors do not make absolute scale observable in every arbitrary scene. Explain the ambiguity before promising distances in metres. The diagram shows the same angular size for two different size-distance combinations. Relative predictions can capture ordering or shape while leaving scale unknown. Some methods, including MiDaS-style training, handle scale and shift ambiguity in an inverse-depth representation. Do not generalize that all relative methods have exactly the same invariance. Metric evaluation needs the camera convention, physical units, and any permitted alignment stated explicitly.

Use four persistent families: global, set, dense, query-conditioned. The first three describe output organization. The fourth adds conditioning and can produce a set or dense mask. Keypoints are structured geometry: K named coordinates per person, and a set across people. Query conditioning is an additional design choice, not a separate tensor layout.

Consolidate the output notation after the visual examples. H and W refer to the chosen output grid, C to semantic classes, N to the number of annotated instances, and K to joint types. The table uses channels last and omits batch size. Detection and instance segmentation have variable target sets even if their networks allocate fixed candidate capacity. The mask implementation extends Part I’s object records. This synthesis appears only after students have seen all unprompted target types visually. Query-conditioned mask notation is introduced in the next section.

Show the point first, then the illustrative dog mask. The click conveys a region of interest without assigning a semantic class name. Both the prompt and the mask are authored teaching elements. This starts a different interface: image plus query produces a spatial answer.

The standard released original SAM interface supports positive and negative points, boxes, and mask inputs. The right panel represents a prior mask; the actual dense mask prompt is low resolution. These inputs can be combined. No class name is required for this interface. The examples are authored, not live SAM runs. Reveal positive point, negative point, box, and previous mask in that order. The negative point lies on the road beside the dog; it marks a region to exclude.