ES 667 · Lecture 5 Computed from the four-point line-fitting example

Learning-rate schedules

A fixed rate makes one compromise for the whole run. First compare the compromise; then watch a decaying rate keep the fast start and settle down later.

A large fixed rate reaches a useful region sooner

Both runs see exactly the same examples in the same order. The only change is the learning rate. We mark the first update where the full-loss gap falls below 0.5.

Controlled comparisonSame dataset, initial parameters, sampled indices, and 80-update window.

larger fixed rateθₜ₊₁ = θₜ − 0.10 gₜ

smaller fixed rateθₜ₊₁ = θₜ − 0.02 gₜ

Full-loss gap during the first 80 updates

fixed η = 0.10 fixed η = 0.02 gap = 0.5

Reveal through update update 0 / 80

fixed η = 0.10

first gap below 0.5not reached

update 8 gap0.386

roletravel quickly

fixed η = 0.02

first gap below 0.5not reached

update 56 gap0.499

trade-offarrives later

The small rate is not wrong; it is slow when distance matters

A large rate gets into the useful region seven times sooner in this run. The next question is whether we must keep taking those large steps after we arrive.

8 vs 56first threshold crossing, not a universal speed ratio

Decay keeps the fast start and makes later moves smaller

Both runs begin at η = 0.10 and therefore make the same first move. In the decayed run, each later gradient is multiplied by a progressively smaller learning rate.

Late-training windowThe chart reveals the final 99 parameter transitions, from update 261 to 360.

fixedηₜ = 0.10

inverse-time decayηₜ = 0.10 / (1 + 0.02t)

Movement near the least-squares optimum

fixed rate decayed rate

Current update update 261 / 360

fixed η = 0.10

η on current update0.1000

distance revealed0.000

mean gap, final 1000.00410

inverse-time decay

η on current update0.0161

distance revealed0.000

mean gap, final 1000.000201

The sampled indices are unchanged; the learning-rate policy changes the moves

Across the final 99 transitions, the fixed-rate path travels 3.794 parameter units. The decayed path travels 0.520 and has about 20 times smaller mean full-loss gap.

14%the decayed path travels 14% as far late in training

Exact experiment. Example A uses x = [−1, 0, 1, 2], y = [−1, 1, 3, 6], and θ₀ = (2, −1). The 360 sampled indices are the exact output of numpy.random.default_rng(12).integers(0, 4, 360).

Checking deterministic results…