EWMA + Momentum Explorer

Memory

Previous summary retained, β 0.90

Keep 90% of the previous memory; add 10% of today’s reading.

Signal

Noise σ 4.4°C

Show constructed slow trend

Add a sudden change to test lag

Change begins day 65

Change size +7.0°C

Before day 1 m₀ = 0

First reading m₁ = x₁

Choice used in this signal plot: seed the smoother at the first observation, then apply mₜ = βmₜ₋₁ + (1−β)xₜ from day 2 onward. The momentum section instead applies its recurrence immediately from m₀ = 0.

The same 120 readings, one adjustable memory

daily reading constructed trend EWMA through today

Click the plot or use ← → to choose a day

before day 1 · m₀ = 0

One number is carried forward

m stands for memory. m_t is the running summary after reading day t; m₀ is its state before any reading.

Retained memory —

Today’s contribution —

New memory —

Use memory to filter a changing gradient direction

In this computed ravine, one coordinate is one hundred times steeper. Plain gradient directions swing across the valley. Momentum blends successive gradients before moving.

Actual Lecture 5 example θ* = (1, −0.5) L(θ) = 0.5(θ₁ − 1)² + 50(θ₂ + 0.5)² gₜ = (θₜ,₁ − 1, 100(θₜ,₂ + 0.5)) mₜ₊₁ = βmₜ + (1−β)gₜ, m₀ = (0, 0) β = .25; θₜ₊₁ = θₜ − ηmₜ₊₁, η = .0195

Same direction: build speed

Plain GD uses only today’s gradient, so its step shrinks as the remaining error shrinks. Momentum keeps part of earlier, agreeing gradients—like repeated pushes in the same direction.

actual θ₁ moves: GD +.0468 → +.0459 · momentum +.0351 → +.0434

Direction reverses: brake the zigzag

Across the valley the gradient flips sign. Old and new contributions oppose each other, so the stored direction becomes much smaller.

actual update 2 in θ₂: plain GD +1.5746 · momentum +.2642

Both methods use the same learning rate, η = .0195. Momentum starts with a smaller move because m₀ = 0 gives m₁ = .75g₀. From update 2 onward, it blends old and new directions. The rolling-ball picture is an intuition, not a literal force law.

Before the first gradient θ₀ = (−1.4, .35) · θ* = (1, −.5) · m₀ = (0, 0) m means gradient memory. We start it at zero because no earlier gradient exists—not because the current gradient is zero.

Parameter space and model predictions move together

path so far measured descent direction −gₜ retained −.25mₜ sum −mₜ₊₁ actual move −.0195mₜ₊₁

Start before update 1

first update · step 1 of 6

Start θ₀ = (−1.4000, .3500); m₀ = (0, 0)

Measure —

Keep —

Add —

Combine —

Move —

The first two updates, without skipping a line

The first update creates memory. The second shows what cancellation means numerically.

Update 1 · start from m₀ = (0, 0)

g₀ = (−2.4, 85)

m₁ = .25(0, 0) + .75(−2.4, 85) = (−1.8, 63.75)

θ₁ = (−1.4, .35) − .0195(−1.8, 63.75) = (−1.3649, −.893125)

Update 2 · remeasure at θ₁

g₁ = (−2.3649, −39.3125)

m₂ = .25(−1.8, 63.75) + .75(−2.3649, −39.3125) = (−2.223675, −13.546875)

θ₂ = (−1.3649, −.893125) − .0195(−2.223675, −13.546875) = (−1.32153834, −.62896094)

Along the valley: the θ₁ gradient components stay negative, so their memory reinforces. Across the valley: .25(63.75) = 15.9375 and .75(−39.3125) = −29.484375 partly cancel, leaving m₂,₂ = −13.546875 rather than another large swing.

Reveal the computed trajectory one update at a time

Rows appear only after the stepper reaches them. Loss is evaluated at θₜ, before that row’s move.

t	θₜ	mₜ (before)	gₜ	mₜ₊₁	θₜ₊₁	L(θₜ)