From pixels to a trained image classifier.

Convolutional
neural networks

Read an image. Build a local detector. Train the whole network. Then look inside it.

Input & geometryLearned weightsResponses & predictions

The route through this lecture

One sequence, following the ML course lecture.

A category has many appearances.

A useful training set must expose variation within a class.

Pose, lighting, scale, background, and occlusion change the pixels.

The class label can stay the same.

A model needs examples of the variation we expect it to handle.

ImageNet made data part of the story.

Categories, many examples, and a shared evaluation task.

What exactly is being predicted?

One image enters. A score for every candidate class leaves.

LeNet → AlexNet → deeper networks

The operation stayed simple; the scale and architectures changed.

An image is an array of numbers.

Spatial axes and channels describe different things.

Grayscale: H × W × 1
RGB: H × W × 3

A channel is a plane of values. Three channels at one location describe one colour pixel.

Could we connect every pixel to 100 hidden units?

First count the weights of the obvious dense layer.

Flattening does not delete the pixels.

It changes the indexing. The architecture determines which relationships are easy to learn.

An ear is useful wherever it appears.

Should the detector need a new set of weights at every location?

A dense layer can learn this.
Why make it learn the same thing twice?

A useful edge may appear at many locations. Make nearby pixels interact, then reuse the same weights at every location.

Flattening is reversible; it does not destroy pixel values. A dense layer simply does not enforce the spatial structure; it does not assume statistical independence of the pixels. Locally connected count includes 9 weights + 1 bias at every padded output.

Local patterns can be composed.

A larger pattern can depend on several smaller ones.

Begin with the ML course’s 6×6 image.

Three bright columns meet three dark columns.

Compare the left side with the right side.

A 3×3 kernel can ask one local question.

First patch: all three columns are bright.

Predict the first output value.

Move one column to the right.

The same weights now straddle the boundary.

One local calculation makes a feature map.

Four legal starts along each axis give a 4×4 result.

One patch becomes one response.

Choose an output cell to move the kernel. Click input pixels to flip 0 ↔ 1.

Reverse the contrast. Reverse the response.

Dark → bright and bright → dark carry different signs.

The exact same calculation in PyTorch.

Name the batch and channel axes even for a single image.

x = torch.zeros(1, 1, 6, 6)
x[:, :, :, :3] = 1
k = torch.tensor([[1., 0., -1.]] * 3)
k = k.reshape(1, 1, 3, 3)
y = F.conv2d(x, k)
# y.shape: [1, 1, 4, 4]
# every output row: [0, 3, 3, 0]

The kernel is not flipped.

Deep-learning libraries usually call this “convolution”; the forward operation is cross-correlation.

Indices here use stride 1, no padding, and one channel. Bias is one scalar per output channel, shared over positions.

Change the question by changing the weights.

The computation stays “multiply, sum, move”.

Return to the beach and the buildings.

The original ML tutorial photographs, with the filtering computed live.

Who chooses the filter?

So far, we did. In a CNN, training does.

Count the legal starting positions.

A length-3 window has four valid starts in a length-6 row.

What if we keep applying valid 5×5 filters?

Use the 32×32 example from the ML lecture.

Does every input pixel participate equally?

Count how many output patches contain each input pixel.

Add a border before sliding.

Zero padding makes additional kernel starts legal.

When does the output keep the same size?

Solve the stride-1 equation.

Stride chooses which starts we visit.

A length-3 kernel on length 8, stride 2.

Geometry changes where we ask.

Change one control at a time. The outlined input cells are the actual sampled taps.

Zero padding invents boundary values. Dilation spaces the taps; it does not add kernel weights. Output cells can be selected to inspect each placement.

Count moves, then include the first position.

Apply independently to height and width. For s = 1 and odd kₑ, p = (kₑ − 1)/2 preserves size. “Same” is a padding policy, not a universal p = 1 rule.

Predict all three output shapes.

Keep channels separate from spatial arithmetic.

Dilation spaces the samples inside a kernel.

A 3×3 kernel can span 5×5 while keeping nine weights.

Replace a local window by its maximum.

The first pooling example: 4×4 → 2×2.

Replace the same window by its average.

A different summary retains different information.

ReLU gates responses. Pooling summarizes a window.

Compare max and average pooling with and without ReLU. This signed feature map is illustrative.

Pool each channel independently. Global average pooling instead averages the entire spatial map into one value per channel.

What information did max pooling remove?

Two different patches can give the same pooled value.

RGB is three aligned planes.

A colour image is not three unrelated photographs.

A single filter spans all input channels.

For RGB and a 3×3 spatial window: one filter has 27 weights.

Three input slices. One output value.

Inspect one RGB patch. Change a channel’s kernel scale and watch its contribution change.

Hand-chosen 2 × 2 RGB patch and weights; each channel has its own spatial weights. One bias is added after the channel sum.

One filter gives one output channel.

A second full-depth filter asks a different question.

First add a bias. Then apply a nonlinearity.

The ML lecture’s final building block completes the layer.

Why put nonlinearities between the layers?

Composition of affine maps is still an affine map.

The same principle applies when the matrices have convolutional structure.

“Sixteen filters” means sixteen whole volumes.

PyTorch convention: NCHW. For groups = 1, every filter reads all input channels. Deeper feature channels need not correspond to colors.

Count the bank, not the image.

Input [N, 3, 32, 32]; sixteen 3×3 filters; padding 1.

Now assemble a complete network.

The LeNet-style network from the ML lecture.

Q1 · What is the input?

Read the axes before reading the architecture.

Q2 · How does 32 become 28?

The first convolution is valid, stride 1.

Q3 · How many parameters are in Conv1?

Six filters; one input channel; one bias per filter.

Q4 · How does 28 become 14?

A 2×2 pool with stride 2 works independently in all six channels.

Q5 · How does 6×14×14 become 16×10×10?

Each of the sixteen filters spans all six input channels.

Q6 · Pool the second set of feature maps.

A 2×2 pool with stride 2 changes space again.

Q7 · How do the feature maps meet an MLP?

Flatten the final activation tensor into one vector per image.

Where are most of the parameters?

Count the dense head one layer at a time.

Revisit the LeNet exercise.

Switch input size. Predict which parameter count changes before you run it.

Valid 5 × 5 convolutions; 2 × 2, stride-2 pools; biases included. This is the fully connected LeNet-style variant in the course materials.

The notebook uses 28×28 MNIST directly.

The same kernels produce a different flatten dimension.

Define the trainable parts.

Five small module definitions implement the notebook network.

self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
self.fc1 = nn.Linear(16 * 4 * 4, 120)
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, 10)

Write the forward pass in the same order.

Each line corresponds to one step we already calculated.

x = F.max_pool2d(F.relu(self.conv1(x)), 2)  # 6×12×12
x = F.max_pool2d(F.relu(self.conv2(x)), 2)  # 16×4×4
x = x.flatten(1)                         # 256
x = F.relu(self.fc1(x))                  # 120
x = F.relu(self.fc2(x))                  # 84
logits = self.fc3(x)                     # 10

Make the training experiment explicit.

Reproduce the ML notebook’s data split and model.

Adam · learning rate 0.001 · batch size 64 · 10 epochs · seed 0

One minibatch has four axes.

Sixty-four grayscale images, each 28×28.

for images, labels in train_loader:
    logits = model(images)     # [64, 10]
    loss = F.cross_entropy(logits, labels)

The training loop is the one you already know.

Convolution changes the architecture, not the gradient-descent recipe.

model.train()
for images, labels in train_loader:
    optimizer.zero_grad(set_to_none=True)
    logits = model(images)
    loss = F.cross_entropy(logits, labels)
    loss.backward()
    optimizer.step()

Did training make progress on unseen validation images?

Measured cross-entropy from the reproduced experiment.

Read the test result in its experimental context.

The fixed 10-epoch model is evaluated on the official test split.

Choose a digit. Inspect its prediction.

Every probability comes from a real forward pass in this browser.

What did the first six filters learn?

Compare random initialization with the trained kernel bank.

Open one calculation inside a learned filter.

At spatial position [8,8], multiply 25 pixels by 25 learned weights.

Conv1 produces six 24×24 response maps.

These are the signed values before ReLU.

ReLU keeps the positive responses.

No shape change: six 24×24 maps remain.

Pool each channel independently.

The six 24×24 maps become six 12×12 maps.

Conv2 combines information across the six channels.

Sixteen filters produce sixteen 8×8 maps.

The second ReLU changes values, not axes.

Sixteen 8×8 feature maps become nonnegative.

After the second pool: sixteen 4×4 maps.

We have reached the feature tensor that feeds the dense head.

Flatten preserves those 256 numbers.

Name the index ranges rather than treating flatten as a learned operation.

The dense head learns combinations of features.

256 activations → 120 hidden units → 84 hidden units.

Ten scores, then one probability distribution.

The output layer has 84×10 + 10 = 850 parameters.

Compare with the original notebook’s feature-map walkthrough.

The same sequence was already a strong part of the ML course.

Original notebook · Reproduction source in this repository

Reuse the kernel twice.

The DL lecture’s exact one-dimensional example.

The loss sends one upstream derivative to each output.

Suppose ∂L/∂y = [1, 2].

One shared weight has several paths to the loss.

The contributions add.

One kernel. A sum of gradient contributions.

Use x = [2, 1, 3], w = [0.5, −1], b = 0. The rest of the network sends upstream gradients g = [1, 2].

Input gradients accumulate too: overlapping windows send multiple contributions to the same input pixel.

The overlapping input also receives two contributions.

The middle value x₁ appears in both windows.

Use the gradient for one update.

Start at w = [0.5, −1], b = 0. Choose learning rate 0.1.

The two-dimensional rule is the same sum.

For stride 1 and no padding, with upstream map G:

Add another sum over examples in the batch. With input channels, each kernel slice uses its corresponding input plane.

Make a whole training step small enough to inspect.

A second example complements MNIST: six 5×5 bar images.

Can two filters separate horizontal from vertical?

Six 5 × 5 training images: one bar, two orientations, three positions. Watch the weights and prediction change.

Actual full-batch gradient descent in JavaScript, cross-entropy averaged over six images, learning rate 0.4, seed 7. All metrics are training metrics. This tiny exercise does not measure generalization to photographs.

Turn a response map into one feature.

This inspector shares the training widget’s current model.

A response is not a probability.

The classifier weights combine both features. Then softmax turns two logits into a distribution.

Uses the current trained model and the same selected example as the preceding trace. Displayed products are rounded; calculations use full precision.

A complete model needs very little code.

model = nn.Sequential(
    nn.Conv2d(1, 2, 3),   # [N,2,3,3]
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    nn.Flatten(1),        # [N,2]
    nn.Linear(2, 2),      # logits
)

optimizer.zero_grad()
logits = model(x)
loss = F.cross_entropy(logits, y)
loss.backward()
optimizer.step()

Make the numbers small enough to follow.

The existing DL lecture averages five horizontal bands from a 45 × 45 crop, then thresholds each band at 128.

Band means: 113.83, 112.37,
145.50, 145.78, 109.32

Binary rows: 0, 0, 1, 1, 0

This deliberately simplified image carries the arithmetic. It is not the photograph’s full-resolution pixel matrix.

What does an edge filter respond to?

Select a filter. The right image is computed from the left image’s grayscale pixels.

The photo comes from the existing Oxford-IIIT Pet teaching evidence. These are hand-designed filters, not learned features. Signed responses use a fixed scale: blue is negative, red is positive, white is zero.

Move the input. What should happen to the feature map?

Reuse of the same local rule gives a translation-equivariant operation under suitable conditions.

Shift the input. Does the response follow?

Compare f(shift(X)) with shift(f(X)). Circular boundaries isolate the stride effect.

At stride 1 with circular boundaries, integer circular shifts commute exactly. At stride 2, only shifts aligned to the stride have an integer output-shift counterpart. Finite zero boundaries can also break equality.

Pooling can change when a feature crosses a boundary.

The second input is the first shifted right by one, with zero fill. Its pooled map is not a simple integer shift of the first output.

Global averaging is invariant to permutations of the positions being averaged. A whole CNN is invariant only when its preceding feature transformation and boundaries support that claim. Low-pass filtering before subsampling can reduce aliasing; it is not a blanket guarantee.

Which input pixels can affect this output?

Track both receptive-field width r and the output spacing j.

Conv → pool → conv: follow the same recurrence.

The DL course’s three-layer example.

Depth grows the receptive field.

Select a stack. The highlighted pixels can influence one interior output, away from boundaries.

The theoretical receptive field counts possible influence. Actual influence depends on weights and activations; the effective receptive field can be much smaller.

The same field of view can hide different functions.

Assume C input, hidden, and output channels; stride 1; dilation 1; ignore biases.

Without ReLU, a stack is still linear (affine with biases), but factorization restricts the possible kernels, and finite padding can complicate boundary equivalence. The next ledgers compare a flatten-based head with a global-average head.

A large matrix with repeated entries.

Build a second complete classifier.

This is the DL course’s 32×32 RGB example.

Both convolutions: 3×3, padding 1, stride 1. Pool: 2×2, stride 2.

Conv1: capacity and work are different counts.

3 input channels → 16 output channels, on a 32×32 grid.

Conv2: less space, more channels.

After pooling: 16 input channels, 32 output channels, 16×16 output.

Global average pooling gives one value per channel.

Each 16×16 map becomes its spatial mean.

Write the ledger before training.

Counts include learned biases; MAC total covers only convolution and linear layers. Pooling, averaging, nonlinearities, bias additions, and softmax also cost work.

Shared weights still do repeated work.

Change resolution or channel counts. Predict which totals will move before touching a control.

One image, square 3 × 3 convolution, stride 1, padding 1, groups 1, with bias. MACs count one multiply-accumulate as one unit; exclude biases and activation functions. Float32 storage here is only the output tensor, not total training memory.

Double both spatial dimensions.

Keep channels, kernels, strides, and padding fixed.

What changed when CNNs scaled up?

AlexNet kept the same basic sequence and made it much larger.

VGG: repeat small filters.

Two 3×3 layers can reach a 5×5 neighbourhood.

A 1 × 1 filter is a shared channel mixer.

Make the expensive spatial operation narrow.

Keep 28×28 resolution; map 64 channels to 128.

Did the bottleneck meet the target?

Change the intermediate width and recalculate all three layers.

Try several spatial scales in parallel.

Inception combines different local questions at the same resolution.

Ask the next block to learn a correction.

Preserve a direct route for the current representation.

Try the shortcut with one number.

Let x = 2 and F(x) = ax.

Keep a path for the existing representation.

This is the residual sum before any optional post-addition nonlinearity. The identity derivative shown applies to a shape-matched identity skip.

Depthwise + pointwise factors the operation.

This factorization restricts the kernel family. It is not an exact replacement for every standard convolution, and lower MACs do not guarantee proportional latency savings.

Look inside the real pretrained backbone.

The same pet photograph passes through ResNet-18.

Start from a representation that has learned useful structure.

requires_grad_(False) freezes parameters; eval() changes modules such as BatchNorm and dropout; no_grad() disables graph recording. Freezing alone does not freeze BatchNorm running statistics.

A frozen backbone can become a feature extractor.

This is the ML lecture’s “store activations” transfer-learning recipe.

The pretrained weights come with a preprocessing contract.

Resize, crop, range, and normalization affect the representation.

Freezing, evaluation mode, and no_grad do different jobs.

Three independent controls are easy to confuse.

Compare the actual pet transfer-learning recipes.

Existing DL course evidence: six breeds, 72 train / 36 validation images.

Head only: 33/36 · late stage: 27/36 · all layers: 34/36 at the selected validation checkpoints.

Can we see structure in the learned representation?

Project the same 1,000 test images into two dimensions.

A two-dimensional plot is not the full classifier.

Separation can be suggestive without being a complete explanation.

Make the failure small enough to inspect.

A successful tiny-batch overfit is a plumbing check. It is not evidence that the model generalizes.

Keep the original tutorials within reach.

Each notebook answers a particular question.

One last calculation: explain each axis.

Input [8,3,32,32], Conv2d(3,16,3,padding=1), then 2×2 pool.

Can you explain the network without naming an architecture?

Runnable PyTorch parity companion · Existing convolution notebook · Sources and teaching choices