Interactive Explainer

Active Learning, Picked Live

Pool-based active learning on a 2-D classification problem. Pick an acquisition function, step the loop, and watch the decision boundary tighten where the queries land. Race four strategies against each other on the same dataset and same starting label budget.

~16 min Active Learning · Acquisition · Label Efficiency

Prelude

The labelling-budget problem

You can label 1000 satellite tiles, or 1000 thermal frames, or 1000 sleep epochs — but no more. Which 1000 do you spend the budget on? Picking randomly is the baseline; picking the points the current model finds most confusing usually beats it by a lot. That's active learning.

The slogan. Don't show the model points it already knows the answer to. Spend labelling budget on the boundary it's most uncertain about.

Step 1

The four acquisition functions you'll meet

Random. Pick a pool point uniformly. The honest baseline.
Uncertainty (least-confidence). Pick the point where $\max_c p(y=c\mid x)$ is smallest. The model is "least sure" here.
Margin. Pick the point where the gap between top-1 and top-2 class probabilities is smallest. Useful when uncertainty doesn't separate hard cases from easy near-tie cases.
BALD / committee disagreement. Train a small ensemble (or use Monte-Carlo dropout); pick the point where the committee disagrees the most. Approximates Bayesian information gain.

Step 2

Run the loop on a real pool

We seeded a 2-D classification problem (3 colour-coded classes, with overlap). 5 labels start the model; click Acquire to add one more. Watch the decision boundary tighten exactly where the chosen acquisition pulls a label from. The legend keeps four scoreboards in sync — same starting set, same logistic-regression-with-RBF-features model, four strategies racing on accuracy.

active strategy labels = 5 / 200 test acc —

Pool + chosen strategy boundary

Acquisition score map

Accuracy vs labels (4 strategies)

What to watch for. The acquisition score map (middle) is a heat-map of how informative each pool point is right now. Random ignores it; uncertainty picks the brightest spot; margin picks where two classes nearly tie; committee picks where ensemble members disagree (this often catches the most useful boundary points first). Run 25 acquisitions on each strategy and read off the gap on the right.

Step 3

The traps real labelling budgets fall into

Cold start. With $\le 5$ labels every strategy is mostly random — the model is too uninformed to identify "useful" points. Always begin with a small random seed.
Redundant queries. Pure uncertainty sampling tends to repeatedly pick neighbours of the same ambiguous region. Add a diversity term (k-medoids on the top-$k$, or BatchBALD) to spread queries out.
Bias against rare classes. If a class has very few examples in the pool, uncertainty will ignore it. Use class-balanced sampling or weight rare classes in the acquisition.
Acquisition mismatch. The model class you use for active learning must be the model class you train on labels. Acquiring with logistic regression then training a neural net is a fast way to leave label-efficiency on the table.
Reproducibility. Random seeds matter enormously in AL; report results as a curve over multiple seeds, not single runs. (See practice-side recommendation on multi-seed reporting.)

Step 4

Where active learning is the right tool

Sensor / measurement placement. Pick the next location/sensor/probe by maximum predictive uncertainty under your current field model. Classical Bayesian experimental design — see the BO article for the related expected-improvement framing.
Image / video annotation. Frame-level annotation is expensive; AL targets frames where the current model is least confident. Standard for medical imaging, autonomous driving datasets, surveillance video labelling.
Satellite tile selection. Out of millions of unlabelled tiles, AL surfaces the few hundred that the detector is least sure about, multiplying labelling efficiency.
NLP fine-tuning. Choose which texts a human annotator should label next; particularly valuable for span / entity annotation where each item is slow.
Drug discovery. Pick which molecules to assay next — a wet-lab experiment costs hundreds of dollars per data point, AL is essential.

Final takeaway. Active learning is one of the cheapest ways to multiply labelling budget. It costs only a model retrain per query and a small bit of book-keeping. For any annotation pipeline with a finite budget, it should be the default rather than the optional add-on.

Step 5

Why active learning often fails in deep learning

The literature is full of "active learning works" papers and the production lore is full of "AL didn't beat random sampling for us". Both are right; the reconciling observation comes from a 2021–22 wave of careful empirical work (Munjal et al., 2022; Lüth et al., 2024):

Random is a strong baseline. When your pool is i.i.d. with the test set and your model is well-tuned, random sampling is hard to beat. Most reported AL wins disappear under careful seed averaging.
The model matters more than the acquisition. Switching from logistic regression to a calibrated deep net often moves accuracy more than swapping between BALD and entropy.
Class imbalance is where AL wins. When some classes are 100× rarer, random sampling under-represents them; uncertainty-based AL (especially with class-aware acquisition) recovers them faster.
Bad calibration kills AL. Acquisition functions read confidences. If your model is wildly miscalibrated (see the calibration article), the acquisition function reads garbage.
Batch-mode AL is hard. Single-query AL is well-studied; batch-mode (acquire $B$ samples per round to keep GPUs busy) has been an open problem until BatchBALD (Kirsch et al., 2019) and CORESET-based methods. Use these for any practical pipeline that retrains in batches.

Step 6

Reading list

Settles (2009) — Active Learning Literature Survey. The historical reference.
Gal, Islam, Ghahramani (2017) — Deep Bayesian Active Learning with Image Data. The BALD-with-MC-dropout paper.
Sener & Savarese (2018) — Active Learning for Convolutional Neural Networks: A Core-Set Approach. The diversity-first paper.
Kirsch, van Amersfoort, Gal (2019) — BatchBALD. The batch-mode BALD paper.
Munjal et al. (2022) — Towards Robust and Reproducible Active Learning Using Neural Networks.
Lüth, Bungert, Klein, Maier-Hein (2024) — Navigating the Pitfalls of Active Learning Evaluation.