Learning outcomes

By the end of this lecture you will be able to:

1. Condition a diffusion model on text via cross-attention.
2. Derive the classifier-free guidance update rule and pick .
3. Explain why latent diffusion made Stable Diffusion shippable.
4. Describe DDIM sampling and skip 95% of steps with no retraining.
5. Recognize DiT (diffusion Transformer) as U-Net's replacement.
6. Place flow matching and consistency models as the 2026 frontier.

Where we are

Last lecture · DDPM — forward noise, learn reverse, predict ε. Works on MNIST, toy 2D. But how do we get from there to Stable Diffusion and Sora?

Four questions:

1. How do we condition on a text prompt?
2. What is classifier-free guidance?
3. Why does latent diffusion matter?
4. What about faster sampling — DDIM and DiT?

How to condition a diffusion model

We want — predict noise given a condition (class label, text embedding, image).

Three common ways to inject :

1. Concatenate to the input channels (for class labels).
2. Add to the time embedding (simple but loses detail).
3. Cross-attention between and intermediate features (Stable Diffusion, most modern).

Cross-attention · the painter-following-instructions analogy

The result · a dynamic, spatial link between text and pixels. The "cat" word strongly influences cat-shaped regions; the "background" word influences corners.

Cross-attention · how an image listens to text

The mechanism — for each image region:

1. Get word vectors from a frozen text encoder (CLIP) → keys and values .
2. Get region "questions" from the U-Net's intermediate features → queries .
3. Score relevance · , then softmax.
4. Final instruction · weighted sum of → used to denoise that region.

text_emb = clip_text_encoder("a cat astronaut")  # [1, 77, 768]

class CrossAttention(nn.Module):
    def forward(self, spatial_features, text_emb):
        Q = self.q_proj(spatial_features)
        K = self.k_proj(text_emb)
        V = self.v_proj(text_emb)
        return softmax(Q @ K.T / sqrt(d)) @ V

Cross-attention · worked numeric

Prompt: "a red square". Two word vectors: , .

Top-left pixel's current state: (more square-like than red-like).

Step 1 · scores.

"Square" is much more relevant.

Step 2 · softmax → weights. Suppose , .

Step 3 · final instruction.

Used to update this pixel — slightly more red, much more square.

Cross-attention · why it works for text conditioning

Self-attention inside the U-Net mixes spatial positions. Cross-attention between spatial features and text features is where the prompt enters.

Visualizing the cross-attention maps reveals exactly this — each word has a blob of pixels that listened most to it. This is the mechanism behind DreamBooth, Prompt-to-Prompt, and every prompt-editing technique.

CFG · the geometry

CFG · the two-compass analogy

CFG · derive the formula step by step

1. Generic direction. (null prompt).
2. Prompt direction. .
3. Pure prompt influence.
   
4. Extrapolate by guidance scale :

• → pure unconditional (ignore prompt).
• → exactly the conditional prediction.
• → amplify prompt adherence by overshooting.

Default in Stable Diffusion.

CFG in pictures

Training with CFG-ready dropout

To enable CFG at inference, the training needs both conditional and unconditional examples:

def training_step(x0, prompt):
    t = sample_t()
    noise = torch.randn_like(x0)
    x_t = add_noise(x0, noise, t)

    # Dropout 10% of prompts to null (enables unconditional path later)
    if random.random() < 0.10:
        c = null_embedding
    else:
        c = clip_text_encoder(prompt)

    pred_noise = model(x_t, t, c)
    return F.mse_loss(pred_noise, noise)

Same network learns both modes. At inference, you run it twice and extrapolate. Cost · 2× compute per step. Benefit · any w at generation time.

CFG · vectors in 2D

Worked example · CFG arithmetic at one step

Larger pushes the noise prediction further along the "more cat-like" direction. Subtract that bigger noise → step lands further toward a strongly cat-shaped image.

Picking a CFG scale · practical guide

Why diffuse in latent space · the intuition

Most pixels in an image are correlated. A patch of blue sky has hundreds of near-identical pixel values. Diffusing each one independently is wasteful.

Result · 48× fewer dimensions, same perceptual quality, ~10× faster sampling. This is the single change that made Stable Diffusion runnable on consumer GPUs.

Latent diffusion · the zip-the-file analogy

The dimension math

512×512 RGB = dimensions.

Stable Diffusion's VAE compresses to 64×64×4 = dimensions.

Saving · fewer dimensions for the expensive U-Net to process.

The diffusion loop runs in latent space; the VAE encoder runs once at the start, decoder once at the end. Single biggest reason Stable Diffusion is shippable on consumer GPUs.

Stable Diffusion architecture

Stable Diffusion · annotated full stack

Latent diffusion · three components

Why 1000 steps · the quick math

DDPM's forward process adds tiny Gaussian noise at each of steps. To keep the final distribution close to and each step's noise increment small (so the reverse can be Gaussian too), has to be large — 1000 is the typical choice.

DDIM (next slide) breaks this by reinterpreting the reverse process as non-Markovian, so you can skip steps without retraining.

DDPM is slow · 1000 steps

Vanilla DDPM sampling requires running the U-Net 1000 times per generation. Each step is a forward pass of a ~1B param network.

We want to make it faster without retraining. Two approaches:

• Fewer sampling steps (DDIM, DPM-Solver)
• Better architecture (DiT, replacing U-Net with Transformer)

DDIM · the foggy-hill analogy

The same trained model gets used · DDIM just chooses which subset of timesteps to evaluate. No retraining. 20× speedup at no cost in sample quality.

DDIM · skip 95% of steps

DDIM · deterministic sampling in 20–50 steps

DDIM reformulates the reverse process to be deterministic given the initial noise. Crucially, the trained DDPM model can be sampled with DDIM — no retraining.

Effect · 50 DDIM steps ≈ 1000 DDPM steps in quality. 20× speedup essentially for free.

In 2026, DDIM (and its successor DPM-Solver++) is the default sampler in every diffusion library.

DiT · the global town-hall analogy

DiT · how it works

Peebles & Xie 2023 · replace the U-Net with a Transformer.

1. Patchify the (small, latent) noisy image into a grid of patches (like ViT).
2. Treat patches as tokens · flatten each → vector. Image becomes a sequence.
3. Transformer blocks · self-attention lets every patch look at every other.
4. Re-assemble → noise prediction.

Why · Transformers scale incredibly well. More data + more compute = better, indefinitely. Inheriting LLM-ecosystem optimizations (FlashAttention, tensor parallelism). Backbone of Sora, Stable Diffusion 3, and most 2024+ frontier image/video models.

What ships in 2026

All diffusion-based. All descendants of the 2020 DDPM paper.

Diffusion for non-image modalities

• Text · text-diffusion (still exploratory; LLMs beat it for quality).
• Audio · AudioGen, Riffusion, MusicLM.
• 3D · DreamFusion, Gaussian Splatting integrates with diffusion priors.
• Molecules · RFdiffusion designs proteins (Baker lab, won 2024 Nobel adjacent).
• Robotics · Diffusion Policy (Chi 2023) — action generation via diffusion.

Diffusion is one of the two big generative paradigms now (the other: autoregressive LLMs).

2026 · the sampler menu

DiT · Transformer backbone

Peebles & Xie 2022 · replace U-Net entirely with a Transformer.

Diffusion · by modality in 2026

Consistency models · winding road vs teleporter

Consistency models · the property

Train to satisfy:

for any two points on the same trajectory.

Inference:

1. Sample .
2. Run network once: .
3. Done.

SDXL-Turbo (2023) · 1-step generation at near-50-step DDIM quality. LCM adapters · drop-in for Stable Diffusion. GAN-speed with diffusion-quality, finally.

Flow matching · the next paradigm?

Rectified flow (Liu 2022) and flow matching (Lipman 2022) generalize diffusion:

• Train a network to predict a velocity field from noise to data.
• Sample with ODE solver in 4-10 steps (vs diffusion's 50+).
• Cleaner math; often better quality at fewer steps.

Stable Diffusion 3, Flux, and many 2024+ models use flow-matching instead of pure diffusion. The boundary is blurring — both are "continuous-time generative models."