Input (8x8)
Click a cell to edit its value
Kernel (3x3)
Element-wise Products
Output (6x6)
Dot product at current position:
Press Step or Play to begin.
Output dimension formula
Three things students get wrong about convolution
"Isn't this just matrix multiplication?"
Not quite. Matrix multiplication combines every row with every column.
Convolution is local: an element-wise product between
the kernel and a small input patch, summed to one scalar. The kernel
then slides to the next patch. This locality
(parameters shared across positions) is what gives CNNs translation
equivariance and dramatically fewer parameters than a dense layer.
"The output is always smaller than the input."
Only with zero padding. Crank the padding slider to
P = 1 with kernel size K = 3 and the output is
the same spatial size as the input — "same" padding. Modern
deep CNNs use same padding almost everywhere so the network depth
isn't capped by the input resolution.
"Convolution and cross-correlation are the same thing."
Mathematically they aren't — true convolution flips the kernel.
In deep learning we almost always implement cross-correlation
and call it convolution; the network learns the kernels anyway, so the
flip doesn't matter. The signal-processing literature is stricter
about the distinction.
Where this fits in the CNN story
The kernels you see here are hand-designed — Edge (H/V),
Sharpen, Blur are classics from image processing. In a CNN the network
learns the kernel values from data. The first conv layer typically
ends up learning Gabor-like edge detectors that look uncannily similar to
the ones you're playing with. Visualising the learned filters of AlexNet's
first layer is one of the most-reproduced figures in deep learning for
exactly this reason.
Stride controls downsampling. Stride 2 halves the spatial
dimension at each step — the classic alternative to max-pooling. ResNet-50
uses stride-2 convolutions at the start of each stage instead of pooling.
Try stride 2 above and watch the output shrink by exactly that factor.
Receptive field grows with depth. One 3×3 conv sees a 3×3
patch. Two stacked 3×3 convs see a 5×5 patch.
k stacked 3×3
convs see a (2k+1)×(2k+1) patch with the same parameter count
per layer. This is why VGG was a step change — replacing one 7×7 with
three 3×3s gave the same receptive field at fewer parameters and more
non-linearities. See the
receptive-field article
for a live demo.
Sliding-window cost. A K×K conv on a W×W feature map at
stride 1 does
W² · K² · C_in · C_out multiply-adds. For
ResNet-50 on 224×224 input the first conv layer alone is ~118M MACs —
this is why ImageNet training was a GPU watershed moment.
Reading list
- LeCun et al., 1998 — Gradient-based learning applied to document recognition. The original LeNet paper; still the cleanest introduction to learned convolutions.
- Krizhevsky, Sutskever, Hinton, 2012 — ImageNet Classification with Deep CNNs. AlexNet; first-layer kernel visualisations are the canonical reference.
- Simonyan & Zisserman, 2014 — Very Deep CNNs (VGG). The 3×3-stacked argument for receptive field at low parameter cost.
- He et al., 2015 — Deep Residual Learning. Same-padding everywhere; stride-2 convs replace pooling.
- Dumoulin & Visin, 2016 — A guide to convolution arithmetic for deep learning. The reference for output-size formulas, transposed convolutions, dilated convolutions.