Interactive Explainer

Assigning Anchors by IoU

How does a detector answer “where is the object?” It doesn't, directly. It pre-places thousands of candidate boxes — anchors — and turns each one into a yes/no question: does an object sit here? The labels for that classifier come from one number, the Intersection-over-Union between each anchor and the ground truth. Drag the object below and watch the anchors take sides.

~7 min Deep Learning · Object Detection · Anchors

Prelude

From localization to classification

Regressing a box out of thin air is hard. Anchor-based detectors sidestep it: they tile the image with a fixed set of reference boxes at several scales and aspect ratios, then ask each anchor a much easier question — is there an object overlapping me, yes or no? The overlap is measured by Intersection over Union:

\text{IoU}(A, G) = \frac{\text{area}(A \cap G)}{\text{area}(A \cup G)} \in [0, 1].

Two thresholds turn that continuous overlap into training labels. An anchor is a positive (object) if its IoU is at least $\tau_{\text{pos}}$, a negative (background) if its IoU is below $\tau_{\text{neg}}$, and is ignored in between so ambiguous boxes don't send confusing gradients.

The Lab

Drag the object, label the anchors

The blue box is the ground-truth object — drag it with the mouse or the sliders. Every thin box is an anchor; each grid location carries six — three aspect ratios at two scales. Each anchor is colored positive, ignore, or negative by its IoU with the object, and the best-matching anchor is drawn bold.

Object center $x$ 4.6

Object center $y$ 3.4

Object width $w$ 2.60

Object height $h$ 2.20

Positive threshold $\tau_{\text{pos}}$ 0.50

Negative threshold $\tau_{\text{neg}}$ 0.30

Anchors classified by IoU with the object. Drag the blue box to move the ground truth.

Positive (IoU ≥ τ_pos) Ignore Negative Ground truth

Positive anchors —

Ignored anchors —

Negative anchors —

Best anchor IoU —

Shrink the object small and drag it between grid points. Its IoU with every anchor can drop below $\tau_{\text{pos}}$ — zero positives by the threshold alone. That is why the highest-IoU anchor is always forced positive (drawn bold): without that rule, tiny or oddly-sized objects would have no anchor to learn from and would simply be missed.

The Payoff

Detection is classification plus a nudge

Once anchors are labeled, the detector's job splits into two familiar pieces, trained together:

Classification. For every anchor, predict object vs. background. Positives and negatives supply the labels; ignored anchors contribute no loss. This is the same cross-entropy you use anywhere else.
Box regression. For positive anchors only, predict a small offset $(\Delta x, \Delta y, \Delta w, \Delta h)$ that slides and stretches the anchor onto the ground truth — trained with the smooth-L1 loss from the previous explainer.

Because anchors are dense and come in several shapes, some anchor is almost always close enough to any object to serve as a starting point. “Where is the object?” becomes “which anchors say yes, and how should each nudge its box?”

Thresholds are a design knob. Faster R-CNN's RPN uses $\tau_{\text{pos}}=0.7$ and $\tau_{\text{neg}}=0.3$; its second stage uses $0.5$. Raise $\tau_{\text{pos}}$ and you keep only tight matches (cleaner positives, but fewer of them); widen the ignore band and you throw away more ambiguous anchors. Every anchor-based detector is tuning exactly these numbers.