Assignment 2 · Vision & Sequences

Out after L15 · programming submission + individual post-deadline viva · 13 marks.

Part of being good at deep learning is building models. A bigger part — the one nobody puts on a slide — is figuring out why a model that should work doesn’t. This assignment grades both, across the two big architecture families of the middle of the course. You’ll budget a CNN by hand and train it, measure what ImageNet pretraining is actually worth, hunt down three bugs in a trainer we broke on purpose, and finally build a character-level sequence model and watch its gradients behave (or misbehave) through time.

Goals

Predict tensor shapes and parameter counts before running code (L8: the conv arithmetic).
See, with your own numbers, when transfer learning crushes training from scratch — and when the gap narrows (L9).
Turn the debug ladder into a reflex: symptom → suspect → test → fix (foundations block).
Build a char-level sequence model from scratch and explain, from your own plot, either the vanishing-gradient problem in an RNN or the exponential receptive field of a dilated conv stack (L12–L14).

Setup

Start from ../notebooks/07-cnn-shape-tour.ipynb and ../notebooks/03-debug-ladder.ipynb for parts (a)–(c). Part (d) builds on ../notebooks/10-rnn-by-hand.ipynb (with the minimal ../notebooks/L12/02_char_rnn_language_model.ipynb) for the RNN path, or ../notebooks/L13/01_causal_dilated_conv.ipynb for the TCN path. You’ll want a free Colab GPU for parts (a), (b), and (d); each run below is under ~10 minutes there. Part (c) needs only your eyes and a CPU.

How this is graded (13 marks)

8 marks for the programming submission (parts a–d), 5 marks for an individual post-deadline viva where you defend your shape table, your transfer-learning curves, your three bug reports, and your sequence-model plot, and answer follow-ups. The viva is where understanding is verified — see What the viva will probe.

Part (a) · A CNN on CIFAR-10, shapes first

Design a CNN for CIFAR-10 with at most 500k parameters and at least 3 conv layers.

Before writing any PyTorch, fill in this table by hand for your architecture (use the formula \(\lfloor (n + 2p - k)/s \rfloor + 1\)):

Layer In shape Out shape # params Why

conv1 (k=?, s=?, p=?) 3×32×32 ? ?

…

Include every layer, including the classifier head. Total the parameter column.
Then build it and verify: print sum(p.numel() for p in model.parameters()) and a forward-pass shape trace. If your hand count disagrees with PyTorch, find out why and write one sentence about what you forgot (bias terms are the classic).
Train with standard augmentation (random crop + horizontal flip) for ~15 epochs. Report test accuracy. Anything ≥ 70% is fine — the table is worth more than the accuracy.
One question: your network’s final conv layer sees how much of the input image (receptive field)? Compute it layer by layer, like in the notebook.

Layer	In shape	Out shape	# params	Why
conv1 (k=?, s=?, p=?)	3×32×32	?	?
…

Part (b) · Transfer learning vs from scratch, with starving data

The question: how much is ImageNet pretraining worth, per training example?

Subsample CIFAR-10 train set to {500, 2500, 10000} images (class-balanced).
For each size, train two models for the same number of epochs:
- your Part (a) CNN from scratch;
- a pretrained resnet18 with all layers frozen except a new final linear layer (remember to resize/normalize inputs the way the pretrained weights expect — look up the ImageNet statistics).
Plot test accuracy vs training-set size, one curve per model (6 runs total; each is a few minutes on Colab).
Written (3–6 sentences each):
- Where is the gap biggest, and why? What is the frozen ResNet bringing to the table that 500 CIFAR images cannot teach?
- The frozen ResNet is just a linear classifier on fixed features. Which ML-course model is this exactly? Why does it still beat a deep net trained from scratch at small n?
- Predict (no need to run): at 50,000 images and 100 epochs, which wins? Justify.

Part (c) · The sabotaged script

The script below is a CIFAR-10 trainer that contains exactly three bugs. None of them raise an error — the script runs to completion every time. It just trains badly. Each bug is in a different conceptual category.

Your job, for each bug, is a four-line lab report in this exact format:

Symptom: what you observed (a number, a curve, a printout — evidence, not vibes) Suspect: which ladder rung pointed here and what you hypothesized Test: the minimal experiment that confirmed it (e.g., “overfit one batch”, “print the gradient norm”, “inspect one normalized image”) Fix: the one-line change

Rules of engagement: you may add print statements, plot things, and run the ladder. You may not just diff against a reference trainer and submit “found them”. We grade the diagnosis, not the find — a correct fix with no symptom/test trail gets at most half credit for that bug. (See the LLM policy: asking a model to “find the bugs” defeats the entire exercise.)

# train_cifar.py -- runs fine. trains terribly. three bugs.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

torch.manual_seed(0)
device = "cuda" if torch.cuda.is_available() else "cpu"

train_tf = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.247, 0.243, 0.261), (0.4914, 0.4822, 0.4465)),
])
test_tf = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.247, 0.243, 0.261)),
])

train_ds = datasets.CIFAR10("data", train=True,  download=True, transform=train_tf)
test_ds  = datasets.CIFAR10("data", train=False, download=True, transform=test_tf)
train_dl = DataLoader(train_ds, batch_size=128, shuffle=True,  num_workers=2)
test_dl  = DataLoader(test_ds,  batch_size=256, shuffle=False, num_workers=2)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, 3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, 3, padding=1)
        self.conv3 = nn.Conv2d(64, 128, 3, padding=1)
        self.fc1 = nn.Linear(128 * 4 * 4, 256)
        self.fc2 = nn.Linear(256, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), 2)
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = F.max_pool2d(F.relu(self.conv3(x)), 2)
        x = x.flatten(1)
        x = F.relu(self.fc1(x))
        return self.fc2(x)

model = Net().to(device)
opt = torch.optim.SGD(model.parameters(), lr=5.0, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(10):
    model.train()
    running = 0.0
    for x, y in train_dl:
        x, y = x.to(device), y.to(device)
        loss = loss_fn(model(x), y)
        loss.backward()
        opt.step()
        running += loss.item()
    model.eval()
    correct = 0
    with torch.no_grad():
        for x, y in test_dl:
            x, y = x.to(device), y.to(device)
            correct += (model(x).argmax(1) == y).sum().item()
    print(f"epoch {epoch}: loss {running/len(train_dl):.3f}  test acc {correct/len(test_ds):.3f}")

Hints, in the spirit of the ladder, not the answer key:

Rung 1 says inspect the data after the transform pipeline. What should the per-channel mean and std of a normalized batch be, roughly?
Rung 3 says overfit one batch. If you can’t, the bug is upstream of “not enough data or epochs”. Which two suspects would you test first for a flat (or exploding) loss?
One of the bugs makes each loss.backward() see ghosts of every batch that came before it. What single print statement, placed before opt.step(), exposes it?

After fixing all three, run 5 epochs and report the test accuracy — it should land somewhere sane (55–70%). This confirms you found all three and didn’t introduce a fourth.

Part (d) · A character-level sequence model (L12–L14)

Time to leave images. Build a char-level model on a tiny text corpus and study how information (and gradient) moves along the sequence. Pick one of the two paths below — both train in minutes.

Corpus. Something small and stated: the names.txt from the makemore lineage, a few KB of a public-domain poem, or the first ~50 KB of Tiny Shakespeare. Report which you used and its size in characters.

Baseline (both paths). Compute the cross-entropy of always predicting the empirical unigram character distribution (one pass over the corpus). Your model’s validation loss must land clearly below this “memorize the marginal” baseline — report both, in the same units (nats or bits, consistently).

Path A — RNN from scratch

Implement a char-level RNN cell by hand — \(h_t = \tanh(W_{xh}x_t + W_{hh}h_{t-1} + b)\) plus an output head. No nn.RNN; write the recurrence. Train with backprop-through-time over a fixed window (e.g. 32 characters).
Plot train and val loss; beat the unigram baseline.
Generate ~300 characters by sampling from the model; include the sample verbatim.
Vanishing gradients. For a single BPTT unroll, plot \(\lVert \partial \mathcal{L}/\partial h_t \rVert\) against the time step \(t\). Then add gradient clipping and say what changed.
Questions (2–4 sentences each): why does \(\lVert \partial \mathcal{L}/\partial h_t \rVert\) decay (or explode) as you go back in time — trace it to the repeated multiplication by \(W_{hh}\) and the \(\tanh\) derivative. What does clipping fix, and what does it not fix? In one sentence: how does this vanishing-gradient problem motivate the attention mechanism (L15) you’ll build in Assignment 3?

Path B — Causal dilated conv (TCN)

Implement a stack of causal, dilated 1-D convolutions with dilations 1, 2, 4, 8, … over the character stream. Causal = no peeking at the future — pad on the left only.
Plot train and val loss; beat the unigram baseline; generate ~300 characters and include the sample verbatim.
Receptive field. Derive, layer by layer, how many characters of context the top layer sees as a function of kernel size and the dilation schedule. Verify empirically: zero out an input character far in the past and check whether a given output actually changes at (and only at) the range your derivation predicts.
Questions (2–4 sentences each): why does stacking dilated convs grow the receptive field exponentially in depth, while a plain conv stack grows it linearly? What is the TCN trading away versus an RNN (hint: parallel training vs unbounded memory)?

State your config (hidden size / channels, layers, context length, steps) in one line.

Deliverables

One notebook containing Parts (a)–(b), the three bug reports for (c) in the symptom/suspect/test/fix format with the fixed script (changed lines marked # FIXED:), and Part (d) with its baseline, curves, sample, and gradient/receptive-field analysis.
Restart & Run All must succeed (it’s fine to load saved metrics for the longer runs — note where you did).
A short LLM usage note (see the course LLM policy).

Rubric (internal — sums to 13)

Component	Marks	What we look for
(a) CNN, shapes first	2	Hand-filled shape/param table for every layer, matching PyTorch (or discrepancy explained); ≥70% test acc; receptive field computed layer-by-layer
(b) Transfer vs from scratch	2	Class-balanced subsets; 6 fair runs; accuracy-vs-n plot; names the linear-probe connection; prediction justified
(c) The sabotaged script	2	Three evidence-driven bug reports (symptom → suspect → test → fix), each with a minimal confirming test; fixed script trains sanely (5-epoch acc in range)
(d) Sequence model	2	RNN or TCN from scratch beats the unigram baseline; sample included; the gradient-through-time (Path A) or receptive-field (Path B) analysis done and verified
Programming subtotal	8
Individual viva	5	Defends own shape arithmetic, transfer curves, diagnostic process, and sequence-model plot live; can answer the follow-ups below
Total	13

What the viva will probe

Come ready to, with your notebook open:

Re-derive one row of your shape/param table on the spot, including where the bias terms live.
Say, from your own accuracy-vs-n curves, why the frozen ResNet wins at n = 500 and name the ML model it reduces to.
Walk one bug’s symptom → test → fix trail without reading it verbatim — we may point at a fix and ask “what was the symptom that led you here?”
Explain your own gradient-through-time plot (Path A) or your receptive-field verification (Path B), and connect it to why the next architecture in the course exists.

Common ways to lose points: a shape table that just echoes PyTorch’s numbers (the point is to predict them first); a transfer comparison that isn’t epoch-matched or class-balanced; a bug “fix” with no symptom/test trail; a sequence model that never actually beats its unigram baseline; analysis (written or in the viva) that could have been produced without running your code.