Learning outcomes

By the end of this lecture you will be able to:

1. Explain double descent and when it happens.
2. Pick among L2 / early stopping / augmentation by regime.
3. Apply Mixup / CutMix / label smoothing correctly.
4. Implement dropout (including inverted scaling) from scratch.
5. Choose among BN / LN / GN / RMSNorm by architecture.
6. Explain pre-norm vs post-norm trade-offs for depth.

Recap · where we are

• Architecture — ResNets, He init, ReLU (L2).
• Optimizer — AdamW with warmup + cosine (L4–L5).
• Recipe — debug ladder, error analysis (L3).

Plan for the two sessions

Session 1 · classical & data-centric

1. Double descent — the modern bias-variance picture
2. L2, L1, early stopping (brisk — prereq covered)
3. Data augmentation
4. Mixup & CutMix
5. Label smoothing

Session 2 · architectural
6. Dropout
7. BatchNorm — mechanics, ICS debate
8. LayerNorm — why sequences differ
9. RMSNorm — the modern simplification
10. Pre-norm vs post-norm placement

Two students · the regularization story

Without it · a deep network has more than enough capacity to memorize the training set perfectly while learning nothing transferable. With it · the model is encouraged to find patterns that hold beyond the training data.

What's new in DL regularization vs classical ML

The classical textbook picture

From ES 654 you know the U-curve:

• Too simple → underfit (high bias)
• Too complex → overfit (high variance)
• Sweet spot in the middle

For 50 years ML chose "the middle" via cross-validation. Done.

Q. But modern nets have – parameters for datasets of examples. We should be in catastrophic-overfit territory. Why aren't we?

Double descent · the 2019 surprise

What's actually happening

Three regimes:

1. Classical underparameterized (params ≪ data): U-curve as expected.
2. Interpolation threshold (params ≈ data): test error spikes.
3. Modern overparameterized (params ≫ data): test error drops again.

This is one of the big open questions in DL theory. Prince Ch 20 — "Why does deep learning work?"

Practical implication

You don't need to shrink a model to generalize. You can just go bigger and rely on:

• Implicit regularization from SGD
• Explicit regularization (today's topic)
• Massive data (when available)

ResNet-50 has 25M params; modern LLMs have 10¹¹. Both generalize fine because they are past the interpolation threshold.

L2 · the weight-leash analogy

L2 · derive the gradient

Total loss = data loss + L2 penalty:

Differentiate term by term:

Plug into the SGD update rule :

The extra term is the leash pull toward zero.

Worked numeric · L2 single-weight update

, , , .

The weight ends up slightly smaller — decayed toward zero. Repeat over thousands of steps · big weights shrink unless the data loss really wants them.

L1 vs L2 · the geometry

L2 = weight decay · regrouping the update

Take the SGD-with-L2 step:

Group the two terms:

The factor is slightly less than 1. So at every step, the weight is first shrunk a little (decayed), then updated by the data gradient.

That's why "L2 regularization" is the same thing as "weight decay".

L1 · the sparsity-inducing sibling

Add → encourages many weights to be exactly 0.

DL rarely uses L1 — features are distributed across many weights, not localized in a few. L1's sparsity breaks distributed representations.

Early stopping as implicit regularization

Q. If val loss starts rising at epoch 30, why stop training?

Because continuing means:

• More optimization steps → more capacity effectively used
• Same function class, but walking further into the parameter landscape
• Eventually you'll memorize training noise

Covered in Lecture 1 — the curve with the "best val" marker.

Why augmentation is powerful

A flipped cat is still a cat. A color-jittered cat is still a cat. By training on the augmented versions, the model learns that these transformations should not change the prediction.

Standard vision augmentations

Which augmentations · which problem

Domain	Useful	Avoid
Natural images	flip · crop · color jitter · rotate	vertical flip (changes sky/ground)
Medical imaging	small rotations · mild intensity	flips (mirrors anatomy)
MNIST / digits	small rotations · elastic	flip (6 9)
Satellite imagery	all rotations (no "up") · flip	color jitter (semantic)
Text	synonym · back-translation · mask	random char shuffle

Rule · augmentation must preserve the label. If not, it's noise, not signal.

Advanced · RandAugment, AutoAugment

Instead of hand-picking augmentations:

• AutoAugment (2018) — learn the augmentation policy from data.
• RandAugment (2020) — pick augmentations uniformly; pick magnitude uniformly. Two knobs.

In torchvision:

transforms.RandAugment(num_ops=2, magnitude=9)

The idea

Standard augmentation: one image → one transformed image, same label.

Mixup and CutMix go further: combine two images and interpolate their labels correspondingly.

Mixup and CutMix in one picture

Why Mixup and CutMix work

Three observations:

1. Decision boundary smoothing. The model sees "half-cat half-dog" examples with mixed labels → boundary becomes smoother, not piecewise-constant.
2. Implicit regularizer. Forces calibration — the output distribution has to reflect the interpolation.
3. Free data. No manual annotation; the mixing happens inside the training loop.

Empirically: Mixup/CutMix adds ~1–2% CIFAR-10 accuracy. Essentially a free win for vision.

Mixup · the smoothie analogy

Mixup · the math, step by step

Mix two examples and :

1. Pick a mixing ratio , . Say .
2. Mix inputs. .
3. Mix labels. .
4. Mix the loss equivalently:

Worked numeric · Mixup label

Cats = class 0, dogs = class 1. One-hot:

• (cat)
• (dog)

The model now sees a faded cat overlaid with 30%-opacity dog, and must produce to minimize the loss. Calibrated outputs for free.

Mixup in PyTorch · 10 lines

def mixup_batch(x, y, alpha=0.2):
    lam = np.random.beta(alpha, alpha)
    idx = torch.randperm(x.size(0))
    x_mix = lam * x + (1 - lam) * x[idx]
    y_a, y_b = y, y[idx]
    return x_mix, y_a, y_b, lam

# in training step
x_mix, y_a, y_b, lam = mixup_batch(x, y)
logits = model(x_mix)
loss = lam * criterion(logits, y_a) + (1 - lam) * criterion(logits, y_b)

Hard vs soft targets

Hard target vs soft target · in bars

Why soften the labels?

Label smoothing · derive the formula

Goal · take a tiny fraction of confidence away from the correct class and spread it evenly across all classes.

1. Hard one-hot label for class 3 ():
   
2. Uniform label (complete uncertainty):
   (each )
3. Mix them with weight :
   
   
4. Compute entry-by-entry:
   

Sum is still 1. The compact form:

Why label smoothing helps

1. Prevents overconfidence. Hard labels push logits to → miscalibrated model.
2. Regularizes the output layer. Softer target → smaller logit magnitudes.
3. Label noise robustness. Some labels are wrong anyway — smoothing acknowledges that.

Session 2 · what we will cover

Two architectural regularizers that ship inside the network:

1. Dropout — randomly silence neurons during training.
2. Normalization (BatchNorm / LayerNorm / RMSNorm) — re-center and re-scale activations.
3. Placement — pre-norm vs post-norm.

All 2026-relevant. All in every modern architecture.

The idea (Hinton 2012)

Every forward pass during training:

1. Sample a random binary mask for each hidden unit .
2. Multiply: .
3. Scale surviving units by to keep expected activation the same.

At eval time, turn dropout off — use all units.

Dropout in one picture

Two intuitions for why it helps

Dropout · different masks per pass

Inverted dropout · the part-time-team analogy

Inverted dropout · why divide by

For a single neuron with activation , keep-prob .

Test time: dropout off → output is just . (target expectation)

Training time: output is random:

• with probability : scaled active output
• with probability : dropped, output

Expected output during training:

✓ same as test time. The rest of the network sees the same expected activations in both modes — training and eval behave consistently. This is inverted dropout.

Dropout · worked numeric example

Suppose hidden activations · h = [2.0, 1.5, 0.5, 3.0] and we use p = 0.5 keep-prob.

The training-time scaling-up by is what lets us drop the mask at eval time without changing the network's expected output.

Dropout · the basketball-team analogy

That's what dropout does to neurons · it prevents them from co-adapting (relying too heavily on a few specific neighbors). Each neuron has to become individually useful.

Dropout in PyTorch

self.drop = nn.Dropout(p=0.1)     # typical: 0.1 for Transformers
                                   #          0.5 for MLP hidden layers
                                   #          0.0 for CNNs (usually)

def forward(self, x):
    h = F.relu(self.fc1(x))
    h = self.drop(h)               # apply after the activation
    h = self.fc2(h)
    return h

Where dropout lives in 2026

Hiker in a canyon · why normalization matters

Concretely · BN/LN keep activations centered and unit-scale, which means the loss's curvature in different directions is roughly equal. The optimizer takes confident, direct steps.

Why normalize at all?

Two problems that normalization fixes:

1. Scale drift across layers. Activations grow or shrink with depth. He init addresses this at init; normalization does it at every step.
2. Internal covariate shift (original claim). Distribution of layer inputs changes during training. This explanation turned out to be partly wrong — see next slide.

BN · LN · RMSNorm · the axes

BatchNorm · train vs eval modes

BatchNorm · standardizing exam scores

BatchNorm · worked numeric example

Mini-batch of 4 activations from one neuron: .

Step 1 · mean.

Step 2 · variance.

• Deviations:
• Squared:
• Mean of squares:

Step 3 · normalize. ()

Step 4 · scale + shift with learned :

This vector is what the next layer sees. At eval time · use the running mean/var collected during training, not batch stats. (model.eval() flips this switch.)

The ICS debate

Ioffe & Szegedy 2015 · BN helps by reducing internal covariate shift (ICS) — the changing distribution of layer inputs during training.

Santurkar et al. 2018 — showed ICS was largely a red herring:

BatchNorm in PyTorch

# For 2D / FC: nn.BatchNorm1d
# For 4D / conv: nn.BatchNorm2d

layer = nn.Sequential(
    nn.Conv2d(64, 128, 3, padding=1),
    nn.BatchNorm2d(128),          # ← after the conv, before ReLU
    nn.ReLU(),
)

• Two learnable parameters per channel · (scale), (shift)
• Two buffers per channel · running_mean, running_var
• Initialized to identity ·

When BatchNorm fails

These are exactly the situations that birthed LayerNorm.

LayerNorm · fix for sequences

Normalize across the feature dimension instead of the batch dimension.

No dependence on batch size or other samples.

norm = nn.LayerNorm(d_model)    # applied at every Transformer block

RMSNorm · the cheap modern cousin

Drop the mean subtraction — keep only the scale:

• ~15–30% cheaper than LayerNorm.
• Empirically no worse at convergence.
• Used in Llama, Mistral, PaLM, and most open LLMs from 2023 onward.

# PyTorch 2.4+
norm = nn.RMSNorm(d_model)

The placement matters

Why pre-norm won · the highway analogy

Why pre-norm won · the gradient path

Residual update: . The term is the gradient highway.

Post-norm. Forward: .

• Backward to goes through LN.
• LN's gradient is a complicated, scale-dependent term in the input statistics.
• Gradient on the skip is modulated → can vanish or explode at depth → needs aggressive warmup.

Pre-norm. Forward: .

• has a direct term from the skip.
• LN sits on the side branch; its complicated gradient affects only the sub-layer.
• Highway is clean → trains stable at depth.

Decision table · which norm · which model

Decision rule in two sentences

The stack for a real vision training run

# 1. Architecture regularization
model = ResNet50(dropout=0.0)      # BN already in ResNet

# 2. Optimizer regularization (from L5)
opt = AdamW(model.parameters(), weight_decay=0.05)

# 3. Data augmentation
train_tfm = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.RandAugment(num_ops=2, magnitude=9),
    transforms.ToTensor(),
    transforms.RandomErasing(p=0.25),
    transforms.Normalize(MEAN, STD),
])

# 4. Mixup inside the training loop (conditional)
# 5. Label smoothing in the loss
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)

# 6. Early stopping — checkpoint on val loss