Three-minute summaries of the lectures in ES 667 Deep Learning (2026) at IIT Gandhinagar, followed by a short series on attention. Each one works through a single idea with a small, worked example.
22 videos, in the order of the course.
22 videos
No matches
Try a broader word such as loss, gradient or attention.
Lectures
One summary per lecture of ES 667 Deep Learning (2026), in the order they were taught.
-
Lecture 1
Where do loss functions come from?
Where does a training loss come from? Pick a distribution for the target, let the network output its parameters, and score each setting by the probability it gave the observed data. The negative log of that probability is the loss. The running example is ten coin flips (4 heads, 6 tails), whose likelihood peaks at 0.4, and the video also shows why multiplying 162 probabilities underflows to zero while adding their logs does not.
-
Lecture 2
Where MSE and cross-entropy come from
Mean squared error and cross-entropy come from the same recipe: choose a distribution for the target, then minimize its negative log-likelihood. Gaussian noise on four points from y = 2 + 3x gives squared error, a Bernoulli label gives binary cross-entropy, and softmax over the logits (2, 1, -0.5) gives cross-entropy. The last scene shows why a confident wrong answer costs the most.
-
Lecture 3A
Outliers: how the noise model picks the loss
Twelve points follow a line, except two. Least squares lets those two outliers cut the slope from 0.955 to 0.397. The video shows how the assumed noise model (Gaussian, Laplace or Student-t) decides the loss and how hard each point can pull on the fit. Every number comes from the 12-point example in the lecture notebook.
-
Lecture 3B
Priors: where regularization comes from
After three heads in a row, maximum likelihood says the coin always lands heads. A prior pulls that estimate back to 0.8. MAP adds one term to the usual loss: a Gaussian prior gives L2 regularization and a Laplace prior gives L1, which can set weights exactly to zero. The numbers come from the lecture's coin example and its four-point line fit.
-
Lecture 4A
From weighted sums to neurons: solving XOR
Can one straight line separate the four XOR points? A search over 461,520 lines gets at best 3 of 4 right. The video builds up from a weighted sum to a single neuron, shows why stacking layers without an activation collapses to one linear map, and then solves XOR with two ReLU units using the weights from the slides. It ends by checking that rule on the whole unit square, where it agrees 75% of the time.
-
Lecture 4B
ReLU hinges and universal approximation
How can a network of ReLUs fit a smooth curve? Each ReLU adds one bend. Three bends make a tent, and more bends trace sin x more closely: the worst gap falls from 0.437 with 2 bends to 0.019 with 15. The video then states the universal approximation theorem, lists what it does not promise, and compares width with depth.
-
Lecture 5A
SGD vs minibatch: how much data per step?
Backprop gives you a slope, but measured on how much data? On a four-point line fit, the video compares a full-batch gradient step with SGD and minibatches: each example's own gradient, a sampled step that raises the full loss from 7.125 to 7.205, why the sampled gradient is right on average, and how batch size trades early progress for noise.
-
Lecture 5B
Why gradient descent zigzags: step sizes
The gradient is exact, so why does gradient descent zigzag? In a two-feature regression where one feature is ten times larger, the loss becomes a narrow ravine and one learning rate has to serve a steep and a shallow direction. At rate 0.0195 the path crosses the valley floor on all 32 updates. Rescaling the feature removes the ravine.
-
Lecture 6A
Backprop by hand: one local rule
The loss is 9: which way should w, x, b and y each move? This 3-minute visual summary of Lecture 6 breaks L = (wx + b − y)² into a graph of four small steps, runs it forward, then walks it backward with one local rule, downstream = upstream × local, to get all four derivatives in a single sweep. Along the way it shows why branches add, checks the answers against the symbolic formula and finite differences, and uses their signs for one gradient-descent step.
-
Lecture 6B
Build a tiny autograd engine from scratch
One call to L.backward() fills in every gradient. This 3-minute visual summary of Lecture 6 rebuilds that call on the same example as Lecture 6A, L = (wx + b − y)² with w = 2, x = 3, b = 1, y = 10: each Value stores its parents and local slopes, a dependency-safe order is reversed, and a short loop multiplies and accumulates with +=. The scratch engine matches PyTorch at every node, and a sigmoid built as one fused op gives the same gradient as the same sigmoid built from four small ops.
-
Lecture 7A
Momentum: averaging away the zigzag
Plain gradient descent zigzags across a narrow ravine because the steep gradient flips sign on every step. This 3-minute visual summary of Lecture 7 builds momentum from an exponentially weighted moving average: same-sign pushes add up, flipped signs cancel, and on the Lecture 5 ravine (same gradients, same learning rate) 55 crossings of the valley floor become 12. It ends with what momentum does not fix: the shallow direction still crawls.
-
Lecture 7B
RMSProp and Adam: a step size per coordinate
On the Lecture 5 ravine, one learning rate has to serve a gradient of 2.4 in one direction and 85 in the other. This 3-minute visual summary of Lecture 7 shows RMSProp dividing each coordinate by a running average of its squared gradients, then Adam adding momentum back and correcting both memories for their zero start. The four optimizers run with the slides' own learning rates, so the path comparison shows how each one works and is not a leaderboard.
-
Lecture 8
Backprop through layers: Jacobians to batches
One small dense layer, z = Wx + b with x = (2, -1), is worked through exactly. The video shows that the layer's Jacobian is W, that backprop sends the gradient back as a vector-Jacobian product, and that the weight gradient is an outer product. It then stacks two examples into a batch, where the shared weights add up each example's gradient, and ends with a small dense, ReLU, dense network.
-
Lecture 9A
Valid outputs by design: softplus to softmax
A model often needs a value from a legal set: a positive standard deviation, a probability, a probability vector or a covariance matrix. The video starts from one gradient step that makes a standard deviation negative (0.1 goes to -0.4), then fixes it by learning a raw number and mapping it with softplus, sigmoid, softmax or a triangular factor L. It also shows why the stable forms of softplus and softmax avoid overflow in float32.
-
Lecture 9B
Learning-rate schedules: warmup, decay, cosine
The running example is one seeded run of noisy SGD on the bowl L = x^2/2 + 3y^2, with the same 36 noise draws for every schedule. A constant rate of 0.20 gets close fast and then keeps bouncing, a rate of 0.04 is slow, and a cosine schedule from 0.20 down to 0.04 does both jobs. The video then covers step, exponential, inverse-time and cosine decay, warmup, where scheduler.step() goes in a PyTorch loop, and ReduceLROnPlateau.
-
Lecture 10A
Overfitting, weight decay and early stopping
A CIFAR-10 model reaches 99.6% accuracy on its training images and 44.2% on images it has not seen. The video reads the learning curves, fits a degree-9 polynomial through ten points (training error zero, validation error 1.70), and then shows two fixes on that same example: weight decay, which cuts validation error to 0.065, and early stopping at step 5,000.
-
Lecture 10B
Dropout, augmentation, Mixup and label smoothing
Random crops and flips raise a CIFAR-10 model's test accuracy from 43.0% to 54.6%. The video covers where else a regularizer can act: augmentation that keeps the label (a flipped "b" becomes a "d"), Mixup, dropout and why it needs eval mode, and label smoothing with targets 0.92 and 0.02. It ends with the measured comparison from the lecture's CIFAR-10 runs.
Attention series
How a language model predicts the next token, and how attention works. The interactive lessons go with it.
-
Part 1
Next-token prediction from scratch
How does a language model continue text? We build the whole idea in 3 minutes, one character at a time, using a tiny model trained on 6,478 Indian given names.
-
Part 2A
Why attention?
Two sentences end with the same four words but need different next words. The video tries three simpler fixes first: read only the last token, concatenate a fixed window, or average everything that came before. Then it treats attention as a soft search, where each token scores the earlier tokens and mixes what they offer. The numbers come from the small hand-designed model in the interactive lesson.
-
Part 2B
Query, key, value
Follow one ambiguous word, "bank", through a single attention step. Its query scores every earlier key, softmax turns the scores into weights (0.41 on "river"), the weights mix the values into a message, W_O projects the message back to four numbers, and the result is added to bank's original row. In the cheque sentence the same starting row ends up leaning toward finance. The numbers come from the hand-designed model in the interactive lesson.
-
Part 2C
From attention to prediction
How does attention turn into a next-word prediction? The causal mask comes first, including how much weight (0.67) would leak to later tokens without it. Then the final "the" goes through its query, weights, message and update, and the prediction head gives water 0.367 after the river sentence and teller 0.521 after the cheque sentence. The video ends with the matrix form and with how generation and training use the same computation.
-
Part 2D
Position and order
Attention by itself ignores word order: swap "Maya helps Ravi" to "Ravi helps Maya" and the weighted sum stays the same. This video covers the ways models add order back: position rows added to each word, learned position tables and their length limit, sinusoidal positions, relative-position biases, ALiBi and RoPE. Each one comes with a small worked example from the interactive lesson.