# From fitting a distribution to the VAE: speaker notes
Planned main route: 145 minutes. Timing includes student responses and the specified short demonstrations. Optional extensions are outside this budget.
## Pacing
| Episode | Minutes |
|---|---:|
| Lecture map | 1 |
| Part 1 · Why generate? | 16 |
| Part 2 · Fit a distribution, then sample | 24 |
| Part 3 · Images: PCA and Gaussians | 20 |
| Part 4 · Autoencoders | 20 |
| Part 5 · Can an autoencoder generate? | 18 |
| Part 6 · The variational autoencoder | 46 |
## contents: Lecture 2 at a glance: six parts, twenty-seven sections
**Build 1, ask:** What does each part add to the one before?
**Expected:** A harder kind of data, or a better way to compress and sample it.
**Transition:** Part 1: things that do not exist.
## faces: Who are these people?
**Build 1, ask:** Who are these people? Would you have known?
**Expected:** Students guess celebrities or stock photos. None exist.
**Build 2, ask:** What did the network need, to make these?
**Expected:** Many real photographs to learn from, and a way to turn random numbers into new ones. That is today’s subject.
**Transition:** A smaller test, on the digits we will use all lecture.
**Source / evidence:** Faces generated by StyleGAN: Karras, Laine and Aila, “A Style-Based Generator Architecture for GANs”, CVPR 2019. NVIDIA, CC BY-NC 4.0.
## realfake: Real or generated? Six of these twelve digits are fake
**Build 1, ask:** Which six are fake? Write the positions.
**Expected:** Answers vary; take a quick show of hands for two or three digits.
**Build 2, ask:** Which fakes gave themselves away, and how?
**Expected:** Broken or doubled strokes and blurry edges give some away; the cleanest fakes are hard to tell.
**Point here:** Real digits: six random MNIST test images not used elsewhere (seed 77). Fakes: the first six draws z ~ N(0, I) through the 16-number VAE of Part 6 (seed 78), not chosen by hand.
**Transition:** Generative models are everywhere now; a quick tour, then the common idea.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## llm: A chatbot is a generative model: one prompt, three different answers
**Build 1, ask:** If we send the same prompt again, do we get the same sentence?
**Expected:** No: the model samples, so each run differs.
**Build 2, ask:** Why would we want different answers each time?
**Expected:** Variety: one prompt has many good answers. A model that always gave the single most likely sentence would be repetitive.
**Transition:** How does it choose? One word at a time.
**Source / evidence:** Generated for this lecture on 21 October 2026 with Gemini 3.5 Flash, temperature 1 (assets/examples/provenance.json).
## next-word: Under the hood: a probability for every possible next word
**Build 1, ask:** Which word would the model pick if it always took the most likely one?
**Expected:** “wet”: 17%, but 83% of the time something else is sampled.
**Build 2, ask:** What is the model a distribution over?
**Expected:** The next word given the words so far. Chaining them gives a distribution over whole sentences.
**Point here:** Probabilities are real: SmolLM2-360M-Instruct, computed locally. We do not train language models in this course; the point is only that generation = repeated sampling.
**Transition:** Pictures: one prompt, many images.
**Source / evidence:** SmolLM2-360M-Instruct (Hugging Face), next-token probabilities and three samples at temperature 1, computed for this lecture.
## t2i: Text to image: one prompt, four samples
**Build 1, ask:** What stayed the same across the four, and what changed?
**Expected:** Same: camel, salt flat, sunrise, watercolour. Changed: rider, framing, sky.
**Build 2, ask:** Where does the variety come from?
**Expected:** From random numbers fed into the model; the prompt only narrows what is plausible.
**Transition:** Images can also be the input.
**Source / evidence:** Generated for this lecture on 21 October 2026 with Gemini 3 Pro Image (assets/examples/provenance.json).
## img2img: Image in, image out
**Build 1, ask:** Is there one right answer for “photo of this sketch”?
**Expected:** No: colour, texture and lighting are not in the sketch.
**Build 2, ask:** So what is the model a distribution over?
**Expected:** Over images, given the input image: p(photo | sketch).
**Transition:** Sound works the same way.
**Source / evidence:** Isola et al., pix2pix (CVPR 2017); Park et al., SPADE (CVPR 2019); Zhu et al., CycleGAN (ICCV 2017). Attributed teaching excerpts; copyright with the authors.
## speech: Sound: text in, 24,000 numbers per second out
**Build 1, ask:** What exactly does the model output?
**Expected:** About 191,000 numbers: a waveform sampled 24,000 times a second.
**Build 2, ask:** Is there one right way to say this sentence?
**Expected:** No: speed, pitch, pauses and voice all vary. Another sample would sound different.
**Transition:** Generative models also matter far from art and chat.
**Source / evidence:** Generated for this lecture on 21 October 2026 with Gemini 3.8 Flash TTS, voice “Kore”.
## science: Beyond pictures: generative models in science
**Build 1, ask:** Why would a forecaster want 50 samples instead of one best forecast?
**Expected:** The samples show the range of possible futures: if they disagree, the forecast is uncertain.
**Build 2, ask:** Why not fill a gap in a sensor record with the average value?
**Expected:** The average is smooth and typical-looking but may never happen, and it hides the uncertainty. A sample looks like real data.
**Transition:** Collect the uses.
**Source / evidence:** Price et al., “Probabilistic weather forecasting with machine learning” (GenCast), Nature 2025; Watson et al., “De novo design of protein structure and function with RFdiffusion”, Nature 2023; Zeni et al., “A generative model for inorganic materials design” (MatterGen), Nature 2025.
## uses: What a model of p(x) is good for
**Build 1, ask:** Which of these needs only samples? Which needs a score p(x)?
**Expected:** Create, simulate, augment: samples. Anomalies and compression: a score.
**Build 2, ask:** Which use connects to Lecture 1?
**Expected:** Compression: an event with probability p costs log₂(1/p) bits.
**Transition:** What do all these models have in common?
## recipe: What they share: random numbers in, a learned network, a sample out
**Build 1, ask:** What plays the role of z in the chatbot?
**Expected:** The random choices made when sampling each next word.
**Build 2, ask:** What is the condition for the camel pictures?
**Expected:** The text prompt.
**Transition:** The condition in each example.
## conditions: Conditions steer the sample; today we learn p(x) without one
**Build 1, ask:** What would a class label as the condition let you do with digits?
**Expected:** Ask for a particular digit: “draw me a 7”, with z choosing the style.
**Build 2, ask:** Why start without a condition?
**Expected:** Conditioning adds an input; learning a distribution over images is the same problem.
**Transition:** Two jobs such a model can do.
## two-jobs: A model of p(x) can do two jobs: score and sample
**Build 1, ask:** Which job does the chatbot do when it writes?
**Expected:** Sample.
**Build 2, ask:** Which job does anomaly detection need?
**Expected:** Score: flag inputs with low p(x).
**Transition:** Why not just output the average example?
## why-not-mean: Why not just output the average?
**Build 1, ask:** Would you accept the average as a new handwritten 3?
**Expected:** No: it is a ghost of many 3s.
**Build 2, ask:** When is the average still useful?
**Expected:** As a summary, or a prediction scored by squared error. Not as a sample.
**Transition:** The route for today.
## route: Today’s route: six parts
**Build 1, ask:** Which part will need Lecture 1?
**Expected:** Part 6: the posterior over codes is intractable, so we use the ELBO.
**Transition:** Part 2: the simplest case, one number per example.
## faithful: Old Faithful: 272 eruptions, and the wait before each one
**Build 1, ask:** If you had to invent one new plausible wait, what number would you say?
**Expected:** Most say about 70, the average. Hold that thought.
**Build 2, ask:** What shape does the histogram have?
**Expected:** Two bumps, with a dip near 65–70 minutes.
**Transition:** The task, in three boxes.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## task: The task: examples in, a distribution out, new examples from it
**Build 1, ask:** What do we have at the start?
**Expected:** Only examples, no formula.
**Build 2, ask:** What does “fit” mean here?
**Expected:** Choose the parameters that make the observed waits most probable.
**Build 3, ask:** What does “sample” mean?
**Expected:** Draw new numbers from p(x), for example with torch.distributions.
**Transition:** Why not just use the average?
## mean-bad: The average wait, 70.9 minutes, is a poor sample
**Build 1, ask:** How many waits lie between 66 and 76 minutes?
**Expected:** 46 of 272.
**Build 2, ask:** What would a good generator do instead?
**Expected:** Produce short waits about a third of the time and long waits otherwise, with realistic spread.
**Transition:** Start with the simplest distribution: one Gaussian.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## gauss-fit: One Gaussian: the best μ and σ have a formula
**Build 1, ask:** How many numbers describe this model?
**Expected:** Two: μ and σ.
**Build 2, ask:** Why maximise the log of the likelihood rather than the likelihood?
**Expected:** The log turns a product over 272 waits into a sum, and does not move the maximum.
**Build 3, ask:** Where is the peak of the fitted bell?
**Expected:** At 70.9: in the dip, where there is little data.
**Transition:** Use the fitted bell to score waits.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## gauss-score: Score: how probable is each wait under the bell?
**Build 1, ask:** Which of the three waits does the bell think is most likely? Is that right?
**Expected:** 71: wrong, few eruptions have waits near 71.
**Build 2, ask:** Is a higher average log-likelihood better or worse?
**Expected:** Better: the data are more probable under the model.
**Transition:** Now sample from the bell.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## gauss-sample: Sample from the bell: the draws fill the empty dip
**Build 1, ask:** Before pressing: where will most draws land?
**Expected:** Near 71, the centre of the bell.
**Build 2, ask:** What is wrong with these samples?
**Expected:** Too many waits in the dip, too few in the clumps: one bump cannot make two clumps.
**Notebook demonstration:** Notebook 2.2: the sampling cell draws 272 waits with torch.distributions.Normal(mu, sigma).sample((272,)) and plots both histograms.
**Transition:** Why are there two clumps?
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## two-kinds: The geyser has two kinds of eruption
**Build 1, ask:** How many groups do you see?
**Expected:** Two.
**Build 2, ask:** If you saw only the wait, could you always tell the kind of eruption?
**Expected:** Usually, from which clump it is in; near the dip, not for sure.
**Transition:** Turn that story into a recipe.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## mixture-story: A recipe with a hidden step: pick a bump, then draw
**Build 1, ask:** How many numbers describe this model?
**Expected:** Five: π₁ (π₂ = 1 − π₁), μ₁, μ₂, σ₁, σ₂.
**Build 2, ask:** Name another latent variable from everyday data.
**Expected:** For example: which person wrote a digit, the season behind a temperature, the topic behind a sentence.
**Build 3, ask:** Why a sum over k?
**Expected:** x could come from either bump; the two ways are mutually exclusive, so their probabilities add.
**Transition:** Score one wait under the mixture.
## mixture-score: To score a wait, add over both kinds
**Build 1, ask:** Which kind explains a wait of 66 minutes better?
**Expected:** The short kind contributes slightly more here.
**Build 2, ask:** Why is the short term at 80 almost zero?
**Expected:** 80 is 4 σ above the short bump’s centre.
**Build 3, ask:** Is a difference of 0.23 per wait a lot?
**Expected:** Over 272 waits it is 61 nats: a likelihood ratio of about 10^27.
**Point here:** These are the fitted values from the next section; showing the score first motivates fitting.
**Transition:** How were the five numbers found? The same loop as Lecture 1.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## mix-loss: What we minimise: minus the average log-likelihood
**Build 1, ask:** Which numbers does p(xᵢ) depend on?
**Expected:** The shares π₁, π₂, the centres μ₁, μ₂ and the widths σ₁, σ₂.
**Build 2, ask:** Why average the logs rather than multiply the probabilities?
**Expected:** A product of 272 small numbers underflows; the log turns it into a sum, and the average does not depend on how many waits we have.
**Build 3, ask:** Why the minus sign?
**Expected:** Optimisers minimise by convention; maximising the score = minimising its negative.
**Transition:** What exactly does the loop change?
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## mix-params: The numbers to learn, each kept valid
**Build 1, ask:** What could go wrong if the loop changed σ directly?
**Expected:** One large step could make σ negative, and a negative width is meaningless.
**Build 2, ask:** Why is exp(log_s) always positive?
**Expected:** exp of any real number is positive.
**Build 3, ask:** What does softmax give for w = (0, 0)? For w = (0, 1)?
**Expected:** (½, ½). For (0, 1): (1/(1 + e), e/(1 + e)) ≈ (0.27, 0.73).
**Point here:** Lecture 1 used the same trick for σ when it fitted q by gradient descent.
**Transition:** Now the loop: one line per piece of the formula.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## mix-code: The loop: one line for each piece of the formula
**Build 1, ask:** Which line is the formula for p(xᵢ)?
**Expected:** p_short + p_long: the two terms of the sum.
**Build 2, ask:** How many numbers does the optimiser change?
**Expected:** Six: two in mu, two in log_s, two in w (only five are free, because the shares add to 1).
**Point here:** For many dimensions, compute logs of the terms and combine them with torch.logsumexp: the same value without underflow. The notebook’s EM fit (section 2.5) is the classic faster algorithm.
**Transition:** Watch it run.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## mix-fit: Press play: the bumps separate onto the two clumps
**Build 1, ask:** What should happen to the two bumps?
**Expected:** Move apart, one onto each clump, and narrow.
**Build 2, ask:** Why does the curve flatten at the end?
**Expected:** Near the maximum the gradient is close to zero.
**Transition:** Read off the fitted numbers.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## mix-result: It found the two kinds of eruption without being told
**Build 1, ask:** What does π₁ = 0.36 mean in words?
**Expected:** About 36% of eruptions are followed by a short wait.
**Build 2, ask:** Why is the match with eruption length remarkable?
**Expected:** The fit used only waits; it recovered a real hidden cause from the shape of the data.
**Transition:** Now sample from the mixture.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## mix-sample: Sample: pick a bump, then a time
**Build 1, ask:** What fraction of draws should pick the long bump?
**Expected:** About 64%.
**Build 2, ask:** How is this better than the single bell?
**Expected:** The samples reproduce both clumps and the dip.
**Notebook demonstration:** Notebook 2.4: the sampling cell draws k from Categorical(pi), then x from Normal(mu[k], sd[k]).
**Transition:** Would three bumps be even better?
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## how-many: More bumps always fit the training waits better; new waits decide
**Build 1, ask:** Why can the training score never get worse with more bumps?
**Expected:** A model with K + 1 bumps can copy the K-bump fit (give the extra bump zero weight).
**Build 2, ask:** How would you choose K for a new dataset?
**Expected:** Hold out data, or use a score that penalises extra parameters.
**Transition:** Recap Part 2.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## recap-2: Part 2 in one slide
**Build 1, ask:** Which step needed the hidden variable?
**Expected:** Sampling realistic waits, and fitting the two clumps.
**Build 2, ask:** What will break first when x has 784 numbers?
**Expected:** The number of parameters (covariances), and finding good hidden variables.
**Transition:** Part 3: images.
## mnist: MNIST: 70,000 handwritten digits, 28 × 28 pixels each
**Build 1, ask:** How many pixels does each image have?
**Expected:** 28 × 28 = 784.
**Build 2, ask:** Why keep one running example?
**Expected:** Every model can be checked on the same 3: its numbers, reconstruction and code.
**Transition:** What the computer sees.
**Source / evidence:** MNIST (LeCun, Cortes and Burges), grey levels scaled to [0, 1].
## pixels: Our 3 is 784 numbers between 0 and 1
**Build 1, ask:** How many of the 784 numbers are exactly 0 for a typical digit?
**Expected:** Most of them: the background.
**Build 2, ask:** In Part 2, x was one number. What is x now?
**Expected:** A list of 784 numbers. The same three steps, fit, score and sample, now in 784 dimensions.
**Transition:** What does a random point in that space look like?
## random-points: A random point in that space looks like static
**Build 1, ask:** Would you ever expect a random point to look like a digit?
**Expected:** No: neighbouring pixels in digits are strongly related; random pixels are not.
**Build 2, ask:** What must a model of digits learn, then?
**Expected:** Where in the space digits are, and how likely each region is.
**Transition:** First attempt: fit Gaussians to raw pixels.
## one-model: What we fit: one distribution for all 60,000 training digits
**Build 1, ask:** Are we using the labels 0–9 here?
**Expected:** No: generative modelling here is unsupervised; the model sees only images.
**Build 2, ask:** Why one model instead of ten?
**Expected:** We want a model of handwritten digits in general; ten separate models would need the label to sample (that would be p(x | label), a conditional model).
**Build 3, ask:** What is the simplest way to model 784 numbers?
**Expected:** One distribution per number, each fitted separately, like Part 2’s one Gaussian, 784 times.
**Transition:** Start with one of the 784 positions.
**Source / evidence:** MNIST training set.
## one-pixel: Start with one pixel: position (14, 14), in every training digit
**Build 1, ask:** Why is this pixel blank in the 0?
**Expected:** A 0 is a ring; the centre of the image is the hole in the middle.
**Build 2, ask:** What shape is the histogram?
**Expected:** Two spikes: at 0 (no ink) and at 1 (full ink), with a thin spread in between.
**Build 3, ask:** Is a Gaussian a good fit here?
**Expected:** No: it puts most probability on grey values around 0.5, which are rare. It is the simplest model, and it shows what we will have to fix.
**Transition:** Now every pixel.
**Source / evidence:** MNIST training set. Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## indep: Do it for all 784 pixels, then sample each pixel from its own bell
**Build 1, ask:** Why does the mean image look like a blur, not a digit?
**Expected:** It averages all ten digits on top of each other.
**Build 2, ask:** How many random numbers does one sample use?
**Expected:** 784: one per pixel.
**Build 3, ask:** Why are there no strokes?
**Expected:** Each pixel is drawn without looking at its neighbours.
**Transition:** Are neighbouring pixels really unrelated?
**Source / evidence:** MNIST training set. Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## neighbours: Why it fails: neighbouring pixels move together
**Build 1, ask:** Why is the correlation five pixels away negative?
**Expected:** A stroke through the centre often means white space a little further out.
**Build 2, ask:** What do we need instead of independent bells?
**Expected:** A covariance between pixels: like the 2 × 2 covariances in Lecture 1, now 784 × 784.
**Transition:** Second try: one Gaussian with a full covariance.
**Source / evidence:** Correlations over the 60,000 MNIST training digits. Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## threes-map: Digits vary in only a few ways
**Build 1, ask:** What changes as you move to the right? Down?
**Expected:** Thicker strokes to the right; more upright and rounder downwards.
**Build 2, ask:** How many numbers would you need to describe any 3?
**Expected:** A handful: thickness, slant, width, loop size … far fewer than 784.
**Build 3, ask:** Why would modelling codes be easier than modelling pixels?
**Expected:** A few numbers are easy to fit with Part 2’s tools; the code also captures how pixels move together.
**Point here:** The two numbers are the first two PCA directions of the training 3s; one 3 per cell of a 7 × 5 grid (the nearest to each cell centre). PCA is introduced on the next slides.
**Transition:** Two machines: one that finds the code, one that redraws the image.
**Source / evidence:** MNIST 3s. Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## encode-decode: Encoder and decoder: image → a few numbers → image
**Build 1, ask:** What goes in and out of the encoder?
**Expected:** In: 784 pixels. Out: a few numbers.
**Build 2, ask:** What goes in and out of the decoder?
**Expected:** In: a few numbers. Out: 784 pixels.
**Build 3, ask:** How would you check that a code is good?
**Expected:** Decode it and compare with the original image: the reconstruction error.
**Transition:** PCA, seen in two dimensions first.
## pca-2d: PCA, the simplest encoder and decoder, in two dimensions
**Build 1, ask:** If you could keep one number per eruption, which would you keep?
**Expected:** A mix of length and wait: the position along the main diagonal.
**Build 2, ask:** How would you find the direction of most spread?
**Expected:** The top eigenvector of the covariance matrix.
**Build 3, ask:** What is lost by projecting?
**Expected:** The distance from each point to the line: here only 5% of the spread.
**Build 4, ask:** How many numbers did we save?
**Expected:** Half: two numbers became one.
**Point here:** For standardised data with correlation ρ the directions are (1, 1)/√2 and (1, −1)/√2 with variances 1 ± ρ.
**Transition:** Now do this with 784 numbers.
**Source / evidence:** Old Faithful geyser data (Azzalini and Bowman, 1990), as distributed with R (datasets::faithful): 272 eruptions.
## pca-mnist: PCA on 60,000 digits: a mean image and directions
**Build 1, ask:** What does w₁ seem to capture?
**Expected:** Roughly “a 0-like ring versus a 1-like stroke”: the biggest difference between digits.
**Build 2, ask:** How many numbers per image after encoding?
**Expected:** d, for example 10 instead of 784.
**Transition:** Rebuild our 3 from a few numbers.
**Source / evidence:** PCA of the 60,000 MNIST training digits (eigenvectors of the 784 × 784 covariance). Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## pca-slider: Rebuild our 3 from its first d PCA numbers
**Build 1, ask:** How many numbers until you would call it a 3?
**Expected:** Around 10–20 for most people; 50 looks almost exact.
**Notebook demonstration:** Notebook 3.3: the reconstruction cell prints the squared error for d = 1, 2, 5, 10, 20, 50, 100.
**Transition:** How much of all digits does d keep?
## pca-scatter: Each digit as 2 numbers: PCA’s map of 1,000 test digits
**Build 1, ask:** Which digits are easy to tell apart here?
**Expected:** 0 and 1: they sit at opposite ends of w₁.
**Build 2, ask:** Why do some digits overlap?
**Expected:** Two linear directions keep only 17% of the spread; 4, 7 and 9 differ in details these directions do not capture.
**Transition:** Now generate from PCA numbers.
**Source / evidence:** 1,000 MNIST test digits, 100 per class (seed 2024). Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## gen-steps: To generate, four steps: encode, fit, draw, decode
**Build 1, ask:** What does each dot stand for?
**Expected:** One training digit, described by two numbers.
**Build 2, ask:** Why fit a bell to codes instead of pixels?
**Expected:** Two numbers are easy to fit; 784 dependent pixels were not.
**Build 3, ask:** Is the drawn dot one of the training digits?
**Expected:** No: it is a new code, drawn from the fitted bell.
**Build 4, ask:** What turns the dot into a picture?
**Expected:** The decoder, m + W z.
**Build 5, ask:** Which steps run each time we want a new digit?
**Expected:** Steps 3 and 4.
**Transition:** One sample, by hand.
## pca-sample-worked: Steps ③ and ④ by hand: first a code, then an image
**Build 1, ask:** Why scale ε by 2.26 and 1.93?
**Expected:** Those are the fitted spreads of the codes; the fitted centre is 0, so z = 0 + spread × ε.
**Build 2, ask:** Why does adding w₁ with z₁ = 1.81 make the image look more like a 0?
**Expected:** w₁ adds ink on a ring (teal) and removes it from the middle stroke (red).
**Build 3, ask:** What would a code of (0, 0) decode to?
**Expected:** The mean image: the average digit.
**Transition:** More PCA numbers, then more bumps.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## pca-mog: A mixture of K Gaussians over d numbers: try it
**Build 1, ask:** What happens as K grows? As d grows?
**Expected:** More bumps: cleaner, more digit-like samples. More numbers: sharper, but the mixture needs many more parameters.
**Point here:** Fitted by maximum likelihood with EM (full covariances), on the PCA numbers of the training digits; held-out scores are on the 10,000 test digits and can be compared only for the same d.
**Notebook demonstration:** Notebook 3.4: the mixture cell fits K = 10 bumps to the 20 PCA numbers and samples 16 digits.
**Transition:** What limits this recipe?
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## pca-limits: What limits the PCA recipe?
**Build 1, ask:** Why can a straight-line decoder not draw a sharp 3 from 2 numbers?
**Expected:** It can only add 2 fixed images to the mean; every digit is built from the same two strokes.
**Build 2, ask:** What would a better decoder do?
**Expected:** Turn a few numbers into a sharp digit along a curved surface, not a flat plane.
**Transition:** Part 3 recap.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## recap-3: Part 3 in one slide
**Build 1, ask:** Which step of “compress, then fit” is PCA?
**Expected:** The compression: encode to d numbers, decode back.
**Build 2, ask:** What would improve the samples most?
**Expected:** A decoder that turns few numbers into sharp digits.
**Transition:** Part 4: autoencoders.
## pca-as-ae: PCA is an encoder and a decoder
**Build 1, ask:** Which part is the encoder, which the decoder?
**Expected:** Encoder Wᵀ(x − m); decoder m + W z.
**Build 2, ask:** What does PCA minimise?
**Expected:** The squared reconstruction error, summed over pixels and averaged over images, among all linear maps.
**Transition:** Replace the straight lines with networks.
## ae-arch: An autoencoder: replace both maps with small neural networks
**Build 1, ask:** Why squeeze through so few numbers?
**Expected:** To force it to keep only what matters about the digit.
**Build 2, ask:** What would happen with a 784-number code?
**Expected:** It could copy the input and learn nothing useful.
**Build 3, ask:** What is the target output?
**Expected:** The input itself: the network is trained to reconstruct x.
**Point here:** Hinton and Salakhutdinov (Science, 2006) made deep autoencoders work; the idea goes back to the 1980s. One hidden layer of 400 units follows Kingma and Welling (2014).
**Transition:** The same in PyTorch.
## ae-code: Two small networks in PyTorch
**Build 1, ask:** How many weights does the encoder have?
**Expected:** 784 × 400 + 400 + 400 × 2 + 2 ≈ 315,000.
**Build 2, ask:** Why no activation after the encoder’s last layer?
**Expected:** The code can be any real number; a ReLU would forbid negative codes.
**Transition:** What does it minimise?
## ae-loss: The loss: squared error, summed over pixels, averaged over images
**Build 1, ask:** Which weights does the loss depend on?
**Expected:** Both φ (encoder) and θ (decoder).
**Build 2, ask:** Why sum over pixels but average over images?
**Expected:** The sum is the error of one image; averaging over images makes the loss independent of the batch size.
**Build 3, ask:** Why must the autoencoder do at least as well as PCA, in principle?
**Expected:** With no ReLUs and no sigmoid it could represent PCA exactly; the nonlinearities only add options.
**Transition:** Before training: what comes out?
## ae-untrained: Before training: always look at what the network outputs
**Build 1, ask:** Why is every output close to 0.5?
**Expected:** Small random weights make the pre-sigmoid values near 0, and sigmoid(0) = 0.5.
**Build 2, ask:** Why is the loss as large as 185 when every pixel error is below 1?
**Expected:** It is a sum over 784 pixels: about 0.24 each.
**Build 3, ask:** What would it mean if training ended near 185?
**Expected:** The network learned nothing: a bug, or a learning rate that is far off.
**Transition:** Train it: one step at a time.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## ae-step: One training step: forward, loss, backward, update
**Build 1, ask:** How many updates in total?
**Expected:** 14,070.
**Build 2, ask:** Does the encoder get a gradient even though the loss compares only images?
**Expected:** Yes: the reconstruction depends on the code, which depends on the encoder weights.
**Transition:** Watch the reconstructions form.
## ae-train: Press play: reconstructions sharpen as it trains
**Build 1, ask:** What does the network learn first?
**Expected:** The average digit: a blurry blob that lowers the error everywhere.
**Build 2, ask:** Which digits stay hardest to rebuild from 2 numbers?
**Expected:** Ones that resemble other digits, like 4 and 9, or unusual shapes.
**Transition:** Did it just memorise the training digits?
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## ae-test: Did it memorise? Compare training and test error
**Build 1, ask:** What would memorising look like in these numbers?
**Expected:** Very low training error and much higher test error.
**Build 2, ask:** Why is the test error a little higher?
**Expected:** The weights were tuned to the training digits; new digits fit a little less well.
**Transition:** What if we take out the bends?
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## linear-ae: Take out the ReLUs and the sigmoid: a linear autoencoder rediscovers PCA
**Build 1, ask:** Why does a linear autoencoder end up like PCA?
**Expected:** PCA is the best linear encoder–decoder pair under squared error; training finds it (up to a rotation within the plane).
**Build 2, ask:** So what do the ReLUs buy?
**Expected:** Curved maps: a better reconstruction from the same few numbers.
**Transition:** How many numbers should the code have?
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## bottleneck: Wider codes reconstruct better
**Build 1, ask:** Why do the curves stack this way?
**Expected:** A wider bottleneck keeps more detail of each individual digit.
**Build 2, ask:** Is the 128-number autoencoder better at everything?
**Expected:** At reconstruction, yes. Hold that question for Part 5.
**Transition:** What the errors look like.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## bottleneck-recon: The same 8 test digits through codes of 2, 8, 32 and 128 numbers
**Build 1, ask:** What does the 2-number code lose?
**Expected:** Stroke thickness, slant and small details.
**Build 2, ask:** At what size are you satisfied?
**Expected:** Around 16–32 for most people.
**Transition:** Compare with PCA at every size.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## ae-vs-pca: Autoencoder against PCA, at every code size
**Build 1, ask:** Where is the gap largest?
**Expected:** Short codes: 2 to 16 numbers.
**Build 2, ask:** When would PCA still be the better choice?
**Expected:** When you need speed, no training, or a guaranteed, interpretable linear map.
**Transition:** What are the codes themselves good for?
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## ae-scatter: Where the codes land: the autoencoder with a 2-number code
**Build 1, ask:** Which digits are still mixed?
**Expected:** 4 and 9; some 3, 5 and 8.
**Build 2, ask:** The codes go from about −5 to 22. Is that a problem?
**Expected:** Not for reconstruction. For generation, we will see (Part 5).
**Transition:** Recap Part 4.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## recap-4: Part 4 in one slide
**Build 1, ask:** What did the autoencoder improve over PCA?
**Expected:** Reconstruction from short codes, and better-separated codes.
**Build 2, ask:** What do we need for generation that reconstruction does not give?
**Expected:** A distribution over codes to sample from.
**Transition:** Part 5: can an autoencoder generate?
## gen-question: Keep the decoder, throw away the encoder, feed it random numbers
**Build 1, ask:** What should a random code look like, to give a good digit?
**Expected:** Like the codes real digits get.
**Build 2, ask:** Why might N(0, I) be a poor choice?
**Expected:** Nothing in training told the codes to look like N(0, I).
**Transition:** What N(0, I) means, in two dimensions.
## std-normal: N(0, I) in two dimensions: two independent standard bells
**Build 1, ask:** What does the I in N(0, I) say?
**Expected:** The covariance is the identity: unit variance, no correlation between the numbers.
**Build 2, ask:** Where will almost all random codes land?
**Expected:** Within about 2 or 3 of the origin.
**Build 3, ask:** How far from the origin is a draw of N(0, I) in 16 dimensions, typically?
**Expected:** About 4 (the square root of 16): in more dimensions the draws sit on a shell.
**Transition:** Feed such draws to three decoders.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## ae-random: Random codes through three autoencoders
**Build 1, ask:** Which of the three looks most like digits?
**Expected:** The 2-number one.
**Build 2, ask:** The 128-number model reconstructed best. Why does it generate worst?
**Expected:** Its codes for real digits are nowhere near where N(0, I) puts its draws.
**Build 3, ask:** What would you check to find out?
**Expected:** Plot where real codes are, next to the random codes.
**Transition:** Plot where the codes are.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## overlay: Put the random codes on top of the real ones
**Build 1, ask:** Hover a red dot near the origin, then one in an empty area. What does the decoder draw?
**Expected:** Near real codes: a digit; in empty places: blurs or odd strokes.
**Build 2, ask:** For the 128-number model, where would the red points fall?
**Expected:** Even further from the real codes: in 128 numbers the space is mostly empty.
**Point here:** This is the experiment improvised in the PML 2023 recording of Lecture 25, around 29–31 minutes. The decoder runs in the browser.
**Transition:** Try choosing codes yourself.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## explore-ae: Your turn: three ways to choose a code
**Build 1, ask:** Which of the three ways gives the best digits?
**Expected:** Encoding a real digit.
**Build 2, ask:** Click far from any dot: what do you get?
**Expected:** A smudge or an exaggerated stroke: the decoder was never trained there.
**Point here:** The decoder (2 → 400 → 784) runs in JavaScript from data/nets.json, weights stored in half precision.
**Transition:** Decode a whole grid of codes at once.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## ae-grid: Decode a whole grid of codes
**Build 1, ask:** Which parts of the grid are useful for generation?
**Expected:** Those over the clusters of real codes.
**Build 2, ask:** What does the decoder do far from any real code?
**Expected:** It extrapolates from the nearest digits: exaggerated or broken shapes. It was never trained on codes there, and N(0, I) would not even reach most of the box.
**Transition:** Wider codes make it worse.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## spread-what: How close are codes of similar digits? Measure it
**Build 1, ask:** What would a good code do with two 0s?
**Expected:** Put them close together.
**Build 2, ask:** Is a single pair enough?
**Expected:** No: one pair could be unusual; average over many.
**Build 3, ask:** What ratio would mean the code ignores digit identity?
**Expected:** About 1: two 0s as far apart as a 0 and a 1.
**Build 4, ask:** Why might this matter for generation?
**Expected:** If similar digits are close, the space between them holds similar digits, so random codes nearby make sense.
**Transition:** Now the same measurement for wider codes.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## spread-sizes: Wider codes: similar digits drift apart
**Build 1, ask:** Why do distances grow with more numbers?
**Expected:** More room: with nothing pulling codes together, they spread wherever reconstruction is easiest.
**Build 2, ask:** What would we want instead?
**Expected:** Codes kept near a known region, with similar digits overlapping.
**Point here:** Distances averaged over the 100 0s and 100 1s of the display set. PML 2023 Lecture 25 shows the same trend for one pair of 0s.
**Transition:** Walk between two digits.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## interp: Walk in a straight line from a 1 to a 0
**Build 1, ask:** Why does the 2-number path pass through other digits?
**Expected:** In 2 numbers the clusters are packed side by side; a straight line from 1 to 0 has to cross the 8, 3 and 5 clusters.
**Build 2, ask:** Why might the 16-number path cross empty space?
**Expected:** The straight line between two codes need not stay where real codes are; in 16 dimensions most of the space between them is empty.
**Transition:** What we want instead.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## wishes: What we saw, and what we want instead
**Build 1, ask:** How could we push codes towards N(0, I)?
**Expected:** Penalise codes that stray from it, for example with a KL to N(0, I).
**Build 2, ask:** How could each image own a cloud instead of a point?
**Expected:** Let the encoder output a mean and a spread, and draw the code from them.
**Build 3, ask:** Which tool from Lecture 1 will we need?
**Expected:** Variational inference and the ELBO.
**Transition:** Part 6: the variational autoencoder.
## two-roads: Two roads to the VAE
**Build 1, ask:** Which wishes from the last slide does Road 1 address?
**Expected:** Clouds (wish 2) and the pull towards N(0, I) (wish 1).
**Build 2, ask:** Where does Road 2 need Lecture 1?
**Expected:** The posterior over codes is intractable, so we fit an approximation by maximising the ELBO.
**Build 3, ask:** What is the encoder in Road 2?
**Expected:** The approximate posterior q(z | x): it outputs the mean and spread of q for each image.
**Transition:** Road 1.
## road1-cloud: Road 1: encode each image to a cloud, not a point
**Build 1, ask:** Which parts of the code plane does an autoencoder’s decoder ever see during training?
**Expected:** Only the exact codes of training images.
**Build 2, ask:** What does the decoder learn when clouds overlap?
**Expected:** To draw something sensible everywhere the clouds reach, including between images.
**Transition:** What the network looks like.
## vae-arch: Road 1: the network, with a cloud in the middle
**Build 1, ask:** Why log σ² instead of σ?
**Expected:** Any real number is a valid log variance; σ = exp(½ log σ²) is then positive.
**Build 2, ask:** Why reparameterise instead of sampling z directly?
**Expected:** sample() blocks the gradient path from the loss to μ and σ.
**Build 3, ask:** At test time, which code do we use to reconstruct?
**Expected:** Usually the mean μ.
**Transition:** The encoder on our 3.
## road1-penalty: Road 1: keep the clouds near N(0, I) with a KL penalty
**Build 1, ask:** What is the KL for a cloud with μ = 0, σ = 1?
**Expected:** Zero: the cloud is exactly N(0, I).
**Build 2, ask:** What does the KL do to a code with μ = 5?
**Expected:** About 12.5 per number: a strong pull back towards the origin.
**Build 3, ask:** What does the −log σ² term do as σ → 0?
**Expected:** It grows without limit: tiny clouds are penalised.
**Point here:** Road 1 looks like a hand-made penalty, like ridge regression (squared error + λ‖w‖²). Road 2 shows that this exact sum comes out of maximum likelihood, with a weight fixed by σ.
**Transition:** Two forces on the clouds.
## road1-tug: Road 1: two terms pull in opposite directions
**Build 1, ask:** What does the reconstruction term want the clouds to do?
**Expected:** Shrink (less noise in z) and spread apart (no confusion between images).
**Build 2, ask:** What does the KL term want?
**Expected:** Move to the centre and widen to σ = 1.
**Build 3, ask:** Where does the balance land?
**Expected:** Clouds near the centre, small but not tiny, slightly overlapping.
**Build 4, ask:** Why does sampling now work?
**Expected:** The codes fill the region N(0, I) samples from, and the decoder was trained across it.
**Point here:** A sketch, not data. The β slides (6.6) show the real version: β = 0 is panel one, β = 256 is panel two.
**Transition:** Road 2: start from the generative model instead.
## vae-pgm: Road 2: the VAE as a generative model
**Build 1, ask:** Which circle is observed?
**Expected:** x, the image (shaded). z is never observed.
**Build 2, ask:** What does the arrow mean?
**Expected:** x is drawn given z: through the decoder, then pixel noise.
**Build 3, ask:** How many sets of decoder weights are there?
**Expected:** One, shared by all 60,000 images.
**Point here:** Notation of probabilistic graphical models (Bishop, PRML chapter 8): shaded = observed, box = repeated for each example, a dot outside = a parameter.
**Transition:** How to draw a new image from this model.
## vae-draw: To draw an image, follow the arrows: first z, then x
**Build 1, ask:** Does generating need the encoder?
**Expected:** No: only the prior and the decoder.
**Build 2, ask:** Why is this a 9 and not another digit?
**Expected:** Because of where this z sits: the decoder has learned which region of codes means which digit.
**Build 3, ask:** Why display the mean image?
**Expected:** The noise carries no information about the digit; it only makes speckle.
**Transition:** What the likelihood means in numbers.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## sigma: The likelihood, in numbers: a squared error
**Build 1, ask:** What happens to the first term if σ is small?
**Expected:** Errors cost more: the model is pushed to reconstruct precisely.
**Build 2, ask:** Where have you seen squared error = Gaussian likelihood before?
**Expected:** Linear regression: least squares is maximum likelihood with Gaussian noise.
**Build 3, ask:** Is σ learned here?
**Expected:** No, fixed at 0.2. It can be learned; we keep it fixed for clarity.
**Transition:** Train by maximum likelihood, like Part 2.
## vae-mle: To learn θ, maximise log p(x) for every image: an integral
**Build 1, ask:** Why do we need p(x) at all?
**Expected:** To train θ by maximum likelihood, and to score images.
**Build 2, ask:** Why can we not compute this integral exactly?
**Expected:** The decoder is a neural network; the integral has no closed form, and for 16 numbers a grid is hopeless.
**Transition:** Look at one image: which codes could have made it?
## posterior3: For our 3: which codes could have made it?
**Build 1, ask:** Why is the patch so small?
**Expected:** 784 pixels with σ = 0.2 pin the code down very precisely.
**Build 2, ask:** Which part of Bayes’ rule is hard?
**Expected:** The evidence p(x) in the denominator: the integral from the last slide.
**Build 3, ask:** What is the prior here, and what is the likelihood?
**Expected:** Prior N(0, I) over codes; likelihood p(x | z) from the decoder.
**Transition:** In two dimensions we can still add up rectangles.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## vae-infer: The variational part: an encoder that approximates the posterior
**Build 1, ask:** Why do we need the posterior at all?
**Expected:** Training needs log p(x), and the ELBO replaces it using a q close to the posterior.
**Build 2, ask:** What is “variational” about the VAE?
**Expected:** We approximate the posterior with a member of a simple family (diagonal Gaussians) fitted by optimisation: Lecture 1’s variational inference.
**Build 3, ask:** Why is one encoder for all images a good idea?
**Expected:** One forward pass gives q for any image, instead of a separate optimisation per image (this is called amortised inference).
**Point here:** Kingma and Welling (2014), figure 1: solid = generative model p_θ, dashed = variational approximation q_φ.
**Transition:** Where the KL to the prior comes from.
## elbo-derive: Road 2: where the KL comes from, for one image
**Build 1, ask:** Which distribution does the expectation use?
**Expected:** q_φ(z | x), the encoder’s Gaussian.
**Build 2, ask:** Why can log p(x) leave the expectation?
**Expected:** It does not depend on z; it is a constant inside the average over z.
**Build 3, ask:** Which KL can we compute, and which can we not?
**Expected:** KL(q ‖ p(z)) between two Gaussians has a formula; KL(q ‖ p(z | x)) needs the intractable posterior. Maximising the ELBO shrinks it anyway.
**Point here:** This is Lecture 1’s ELBO derivation, with θ → z and 𝒟 → one image x.
**Transition:** Write the ELBO in terms of the network.
## elbo-meets: The ELBO for one image is Road 1’s loss
**Build 1, ask:** Which term rewards reconstruction?
**Expected:** The expected log-likelihood (the first term).
**Build 2, ask:** Where does 1 / 2σ² come from?
**Expected:** The Gaussian density of each pixel around the decoder’s output.
**Build 3, ask:** What happens to the loss if σ is small?
**Expected:** The squared error dominates: the model is pushed to reconstruct precisely, and the KL matters less.
**Transition:** Check the encoder on our 3.
## q3: Check the encoder: its q for our 3, against the true posterior
**Build 1, ask:** Which q is better, and by how much?
**Expected:** The per-image fit, by 0.68 nats.
**Build 2, ask:** Why is the encoder’s q not perfect?
**Expected:** One network must serve all 60,000 images; it cannot be exactly right for each.
**Transition:** Put the two roads side by side.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## roads-compare: Two roads, one VAE
**Build 1, ask:** Which road tells you how to choose the weight between the two terms?
**Expected:** Road 2: the weight 1/2σ² comes from the assumed pixel noise.
**Build 2, ask:** Which road would you use to explain the VAE to a friend in one minute?
**Expected:** Road 1: an autoencoder with noisy codes pulled towards N(0, I).
**Transition:** Train it: the loss, term by term, with numbers.
## vae-loss: The loss for one image is the negative ELBO
**Build 1, ask:** Which term pulls the codes towards N(0, I)?
**Expected:** The KL term.
**Build 2, ask:** Why is one draw enough?
**Expected:** Over many steps and images the noise averages out; the estimate is unbiased.
**Build 3, ask:** What does the KL do to a code with μ = 5?
**Expected:** Penalises it by about 12.5: strong pressure back towards the origin.
**Point here:** Shape of the loss: data fit + penalty, like ridge regression (squared error + λ‖w‖²). Here the penalty is a KL to the prior, and it falls out of the ELBO rather than being added by hand.
**Transition:** Put in the numbers for our 3.
## vae-worked: One step for our 3, with numbers
**Build 1, ask:** Why is z so close to μ?
**Expected:** σ is tiny (about 0.01): the code of this 3 is pinned down precisely.
**Build 2, ask:** Which term is larger, and what does that say?
**Expected:** Reconstruction, by far: fitting the pixels dominates this model.
**Build 3, ask:** What does backward() update?
**Expected:** The decoder’s θ through the reconstruction, the encoder’s φ through both μ, σ (via z) and the KL.
**Transition:** The whole thing in PyTorch.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## vae-code: The VAE in PyTorch: three changes to the autoencoder
**Build 1, ask:** What are the three changes?
**Expected:** Two outputs per code number, a random draw of z, and the KL term.
**Build 2, ask:** How many lines of the training loop change?
**Expected:** None: only the model and the loss.
**Transition:** How the two terms behave while training.
## vae-curves: While training: reconstruction falls, the KL settles
**Build 1, ask:** Why does the KL not fall to zero?
**Expected:** Then q would equal the prior for every image: the code would carry no information and reconstruction would suffer.
**Build 2, ask:** What would a KL of zero mean?
**Expected:** The code ignores the image: every 3 and 7 gets the same q. We will see this happen.
**Transition:** Now sample.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## vae-samples: Now sample: z ~ N(0, I), then decode
**Build 1, ask:** What changed between the first two rows?
**Expected:** Training made the codes fill N(0, I), so random codes land on digits.
**Build 2, ask:** Why are the VAE samples still a little blurry?
**Expected:** Each pixel is modelled as an independent Gaussian around a mean image; the mean of plausible strokes is soft.
**Build 3, ask:** Which digits are missing from the autoencoder row?
**Expected:** Most of them: random codes reach only the digits whose codes sit near the origin.
**Transition:** Where the codes are now.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## vae-scatter: The codes now fill N(0, I)
**Build 1, ask:** Compare with the autoencoder’s map (Part 5). What changed?
**Expected:** Codes now fill a round region around the origin with no large gaps; scales are about 1, not 20.
**Build 2, ask:** Is the cloud exactly N(0, I)?
**Expected:** Not quite: a little wider (spread 1.7) and slightly off-centre. The KL is a pull, not a constraint.
**Transition:** Try the three ways again.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## explore-vae: Your turn again: the same three ways, now with the VAE
**Build 1, ask:** Click in the gaps between digit groups. What do you see now?
**Expected:** Plausible in-between shapes: the space between digits is filled.
**Transition:** A grid, and a walk.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## vae-grid: Decode a grid of codes: digits everywhere
**Build 1, ask:** Compare with the autoencoder’s grid. What is different?
**Expected:** No junk: the whole region that N(0, I) samples from is used.
**Build 2, ask:** Why are the in-between images plausible now?
**Expected:** Each image owned a small cloud of codes during training, so the decoder learned to draw something sensible everywhere in between.
**Transition:** Section 6.7: a weight on the KL.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## beta-what: Put a weight on the KL: β
**Build 1, ask:** Which value of β gives back the real ELBO?
**Expected:** β = 1.
**Build 2, ask:** What do you predict at the two extremes?
**Expected:** β = 0: codes spread out like an autoencoder’s. Huge β: codes collapse onto the prior and stop carrying information.
**Point here:** PML 2023 uses mean-over-pixels squared error plus β times the KL; its β = 0.001 corresponds to about β = 5 here.
**Transition:** Seven trained VAEs.
## beta: Seven VAEs, β from 0 to 256: try them
**Build 1, ask:** At which β do samples look best? When do reconstructions start to blur?
**Expected:** Around β = 1 to 4. By β = 64 reconstructions blur; at 256 everything is the same blob.
**Transition:** Two extremes worth a closer look.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## beta-zero: β = 0: is it just an autoencoder? (An exam question)
**Build 1, ask:** Is a VAE with β = 0 the same as an autoencoder?
**Expected:** Different by construction (the encoder outputs a distribution and z is sampled), but in practice it learns to make σ ≈ 0 and becomes one.
**Build 2, ask:** What does the KL term do to σ?
**Expected:** It stops σ shrinking: −log σ² grows as σ → 0.
**Transition:** The other extreme.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## beta-collapse: β = 256: every code decodes to the same blur
**Build 1, ask:** Why is KL = 0 the cheapest solution here?
**Expected:** β × KL dominates the loss; reconstruction error is worth less than any information in the code.
**Build 2, ask:** How would you spot collapse during training?
**Expected:** The KL term falls to about 0 and samples all look the same.
**Transition:** Wrap-up.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## arc: The whole lecture in one table
**Build 1, ask:** Which models can both score and sample?
**Expected:** The Part 2 models, the PCA mixtures and the VAE (through the ELBO, a bound). The plain autoencoder can do neither.
**Transition:** What comes next.
## next: Next: can we drop the likelihood altogether?
**Build 1, ask:** Why does a pixel-wise Gaussian encourage soft images?
**Expected:** Squared error favours the mean of plausible images, which is blurry.
**Build 2, ask:** What else could tell a generator its images are wrong?
**Expected:** A classifier trained to tell real from generated: a learned critic.
**Transition:** End of the main route. Optional slides follow.
## sparse: A sparse autoencoder: a KL between two coins (optional)
**Build 1, ask:** Why would we want few active units?
**Expected:** Each unit specialises in a feature; this regularises the autoencoder.
**Build 2, ask:** Where else did we add “loss + penalty”?
**Expected:** The VAE: reconstruction + KL; ridge regression: squared error + λ‖w‖².
**Transition:** Back to the main route.
**Source / evidence:** Ng, “Sparse autoencoder”, CS294A lecture notes; Lilian Weng, “From Autoencoder to Beta-VAE”.
## heldout-mnist: Held-out scores of the PCA mixtures (optional)
**Build 1, ask:** Why can we not compare across rows?
**Expected:** Each row is a density over a different number of dimensions, so their units differ.
**Transition:** Back to the main route.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## em: EM in one slide: the classic way to fit a mixture (optional)
**Build 1, ask:** Where have you seen r_ik before?
**Expected:** It is p(k | x), the posterior over the hidden variable: Bayes’ rule.
**Build 2, ask:** Why is EM a cousin of the VAE?
**Expected:** Both alternate between inferring hidden variables and refitting the model; the VAE uses an approximate posterior q because the exact one is intractable.
**Transition:** Back to the main route.
## full-gauss: Second try: one 784-dimensional Gaussian with a full covariance (optional)
**Build 1, ask:** Why are the samples smoother than before?
**Expected:** The covariance makes neighbours move together.
**Build 2, ask:** What is wrong with one bump? Think of Part 2.
**Expected:** Digits form clusters (one per digit, at least); one Gaussian is Part 2’s single bell again.
**Transition:** Idea: describe each image with a few numbers first.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## pca-variance: How many numbers do we need? (optional)
**Build 1, ask:** Why does the curve flatten?
**Expected:** Later directions describe small details and pixel noise.
**Build 2, ask:** What is a good d for generating digits?
**Expected:** A trade-off: small d is easier to model, large d keeps detail. We will try several.
**Transition:** A map of digits in two numbers.
## pca-gauss: One Gaussian over d PCA numbers (optional)
**Build 1, ask:** Why are the d = 2 samples so blurry?
**Expected:** Two PCA numbers can only describe a blurry digit, whatever their values.
**Build 2, ask:** Why do some d = 50 samples look like two digits on top of each other?
**Expected:** The single Gaussian puts a lot of probability between the clusters of different digits.
**Transition:** Use a mixture over the codes.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## search: Search by code: the nearest digits to a query (optional)
**Build 1, ask:** Why search with codes instead of pixels?
**Expected:** Fewer numbers, and distances that reflect shape rather than exact pixel positions.
**Build 2, ask:** What would make two codes close?
**Expected:** Digits the decoder rebuilds similarly: same identity and similar style.
**Transition:** Codes can also clean images.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## denoise: Denoising: damage the input, ask for the clean image (optional)
**Build 1, ask:** Why does the plain autoencoder reproduce the holes?
**Expected:** It was trained to copy its input, holes and all.
**Build 2, ask:** What does the denoising version have to learn?
**Expected:** What digits look like, so it can guess the missing pixels.
**Build 3, ask:** What does this have in common with dropout?
**Expected:** Both randomly remove inputs during training (denoising autoencoders came first, in 2008).
**Point here:** Both have 16-number codes. Vincent et al., “Extracting and composing robust features with denoising autoencoders”, ICML 2008.
**Transition:** Part 4 recap.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## posthoc: A fix we already know: fit a mixture to where the codes are (optional)
**Build 1, ask:** Do the ellipses cover the clusters?
**Expected:** Roughly: about one ellipse per digit group, some shared.
**Build 2, ask:** Why do these samples look better than N(0, I) codes?
**Expected:** The codes are drawn where real codes are.
**Build 3, ask:** What is unsatisfying about this two-stage approach?
**Expected:** Two separate fits; the codes were never shaped for easy sampling; and there is no single model that scores p(x).
**Transition:** What we would like instead.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## evidence-grid: With 2 numbers we can add up rectangles; with 16 we cannot (optional)
**Build 1, ask:** Why is log p(x) positive?
**Expected:** It is a density, not a probability: with σ = 0.2 per pixel, densities are large.
**Build 2, ask:** Why do grids fail in 16 dimensions?
**Expected:** The number of cells is (points per axis) to the power of the dimension.
**Point here:** The fine grid is 241 × 241 cells over a window of ±0.15 around the posterior. A coarse grid over the whole prior gives 243.96: too coarse to resolve the patch.
**Transition:** Sample codes from the prior instead?
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## naive-mc: Average p(x | z) over random codes? Most draws are wasted (optional)
**Build 1, ask:** Why is the estimate so low with few draws?
**Expected:** Most prior draws decode to a different digit; p(x | z) is astronomically small for them.
**Build 2, ask:** Where should we draw codes from instead?
**Expected:** From near the posterior: codes that could have made this 3. That is what an encoder will provide.
**Transition:** Lecture 1’s tool for an intractable posterior.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## vi3: Fit q for our 3: press play (optional)
**Build 1, ask:** What happens to the ELBO at first? Why?
**Expected:** It wobbles and dips: q is still huge, and most codes it draws explain the 3 badly.
**Build 2, ask:** How big is the final gap?
**Expected:** About 0.02 nats: q is nearly the exact posterior.
**Point here:** Adam, step size 0.05 then 0.01, 64 reparameterised draws per step, 1,200 steps (seed 23). The ELBO plotted is the noisy estimate at each step.
**Transition:** One image took 1,200 steps. Now 60,000 images.
**Source / evidence:** Computed for this lecture: scripts/science.py from the networks trained by scripts/train.py (MNIST, fixed seeds).
## amortise: 60,000 images would mean 60,000 separate fits (optional)
**Build 1, ask:** How many variational parameters would per-image VI need for 60,000 images?
**Expected:** 4 × 60,000 = 240,000.
**Build 2, ask:** What does the encoder output?
**Expected:** Two means and two log-variances per image.
**Build 3, ask:** What is the price of sharing one network?
**Expected:** Its q may be a little worse than a dedicated fit for each image (next slides).
**Transition:** Put the encoder in front of the decoder.