Learning outcomes

By the end of this lecture you will be able to:

1. Describe agents as the perceive-think-act loop with external tools.
2. Understand test-time compute scaling in reasoning models (o1, Claude).
3. Read the residual stream view of a Transformer.
4. Recognize induction heads and sparse-autoencoder features.
5. Articulate the open problems of DL (continual learning, data efficiency, grounding).
6. Identify 2026-2030 research directions worth pursuing.

The final lecture

23 lectures of theory + practice. Today — three threads that define the 2024–2026 frontier, plus a course recap.

Four questions:

1. What are agents and what changed in 2024?
2. What are reasoning models (o1, Claude thinking)?
3. What is mechanistic interpretability?
4. What are the open problems — and where does this course take you next?

Brain in a jar · why agents matter

The LLM keeps its strengths (knowledge, language fluency) and gains the missing piece · action. With tools, the same model that can describe how to book a flight can now actually book one.

From chatbot to agent

An LLM chatbot gives you text. An LLM agent can act — call tools, browse the web, write code, click buttons on a screen.

The ReAct loop

ReAct · annotated with tools

Function calling · how agents work

Modern APIs (Claude, OpenAI, Gemini) support structured tool calling:

tools = [
    {
        "name": "search_flights",
        "description": "Find flights between two cities on a date",
        "input_schema": {
            "type": "object",
            "properties": {
                "from": {"type": "string"},
                "to":   {"type": "string"},
                "date": {"type": "string", "format": "date"}
            }
        }
    }
]
response = client.messages.create(model="claude-4", tools=tools, messages=[...])

The model returns a structured call · you execute the function · feed back the result · loop until done.

Computer use · agents controlling UIs

Claude Computer Use (2024) · the model sees a screenshot, outputs mouse + keyboard actions.

Screenshot → Claude → "move_mouse(320, 450) ; click() ; type('hello')" → screenshot

This is RL-like but trained mostly supervised on human demonstrations and synthetic examples. Opens up:

• Browser automation
• Desktop task completion
• Form filling
• Legacy-app bridging

Chain-of-thought · the 2022 discovery

Q: John has 5 apples. He gives 2 to Mary and buys 4 more. How many?
A: He starts with 5. Gives 2 → 3. Buys 4 → 7. Final: 7.

CoT emerges at scale (~60B params). Below that, adding "think step by step" doesn't help.

Reasoning models · outcome vs process rewards

Reasoning · the training recipe

o1 (OpenAI 2024), Claude extended thinking, DeepSeek R1 (2025):

1. Generate many candidate chains of thought for a hard question.
2. Reward each chain · outcome (final answer correct?) and process (each step logical?).
3. Fine-tune via RL to make high-reward chains more likely.
4. At inference · spend 10×–100× more compute, let the model "think."

Worked example · rewarding chains of thought

Question. A bat and a ball cost $1.10 total. The bat costs $1.00 more than the ball. How much does the ball cost?

Candidate 1 (greedy).

1. Bat is $1.00. 2. Ball = . Answer · 10¢.

• Outcome reward · 0 (wrong — common-sense trap).
• Process reward · low (step 2 based on a wrong assumption in step 1).

Candidate 2 (algebra).

1. , . → . → . Answer · 5¢.

• Outcome reward · +1.
• Process reward · high — every step logical.

The model is trained to make traces like Candidate 2 more likely.

The scaling laws recap · in one figure

Test-time compute · the college-exam analogy

Reasoning models (o1, Claude thinking) are exactly this · same base capability as a "college graduate" model · but allowed to use scratch paper before responding. The scratch paper is internal chain-of-thought that the user never sees.

A new scaling axis

Until 2024, the only scaling axis was training compute. In 2024+ we added test-time compute.

Reasoning · tree search at inference

Reasoning models · benchmarks

Comparable gains on HumanEval, MATH, GPQA. This is an entirely new capability curve.

The problem

A 70B Llama has 70 billion parameters. We trained it, and it works. But if it produces a wrong answer — or a dangerous one — we can't read the weights and know why.

Mechanistic interpretability (mech interp) tries to reverse-engineer specific computations inside trained networks.

Induction head · a discovered circuit

Residual stream · the shared-whiteboard analogy

The Transformer equation just says: read from the bus → add a contribution → write back.

Residual stream · worked numeric

Toy model, . Token enters layer 5 with state:

Step 1 · attention update. Attention reads , looks at other tokens, returns:

(boost features 2 and 3 from related tokens)

Step 2 · FFN update. FFN processes alone, returns:

(slightly boost feature 1, decrease feature 2, clarify feature 4)

Step 3 · sum (the residual update).

Same equation: . The "bus" perspective makes circuits findable — induction heads, IOI task, etc.

Sparse autoencoders · the dictionary analogy

The dictionary is much wider than the residual stream (e.g., 100k features for a 12k-dim residual). Most features are 0 for any input · the active ones become human-interpretable concepts.

Sparse autoencoders · the inverted bottleneck

SAE · how it disentangles superposition

Problem · superposition. A single neuron in the residual stream might fire for "Golden Gate Bridge", "the colour red", AND "Python syntax errors". Confusing.

SAE recipe:

1. Input · residual-stream vector (e.g. ).
2. Encoder · expand to a much wider sparse vector (). , with a sparsity penalty forcing most entries to 0 (typically only 10–50 non-zero per input).
3. Decoder · reconstruct .
4. Train to minimize + sparsity penalty.

Worked numeric (toy). (meaningless — superposition). Trained encoder maps to:

Researchers find all inputs that activate feature 6 → all about the Golden Gate Bridge. Label: "Feature 6 = Golden Gate Bridge".

Anthropic 2024 · SAE on Claude 3 Sonnet → millions of human-readable features. Clamp features → control behavior ("Golden Gate Claude" demo).

Why mech interp matters

Two reasons:

1. Safety · identify and edit circuits that produce harmful outputs.
2. Science · understand what representations language models actually learn, and how.

Safety · why alignment matters

As models become more capable, the cost of misalignment grows:

What you've learned · a final recap

Open problems · the short list

1. Reasoning reliability · even o3 still hallucinates; trust calibration is unsolved.
2. Continual learning · models can't easily update with new facts without retraining.
3. Data efficiency · humans learn from 10³ examples; models need 10¹². Why?
4. Alignment · how do we ensure advanced systems remain beneficial?
5. Interpretability at scale · we barely understand what's inside a 70B model.
6. Multi-modal reasoning · image+text+video+action reasoning jointly is still weak.
7. Grounding · LLMs know about the world through text; they don't have embodied experience.
8. Energy & cost · a GPT-4 query costs ~pennies but society-scale deployment is energy-intensive.

Each is a PhD worth of work. Pick one.

Where's the field going?

Predictions (take with salt):

• Reasoning compute will scale 10× per year for a few years — expect ~1000× by 2028.
• Agentic AI will move from demos to production workflows in 2026.
• Multimodal will absorb audio, video, 3D into the same models.
• Open-weight models will continue closing the frontier gap (Llama, Mistral, DeepSeek).
• Domain-specific small models (medical, legal, scientific) will win niche deployments.
• Safety & alignment research will become mainstream.

The 24-lecture arc

What you can now do

1. Read any 2026 ML paper and understand the architecture.
2. Implement a Transformer, a diffusion model, a LoRA fine-tune from scratch.
3. Know when to use what — CNN vs ViT, SGD vs AdamW, RLHF vs DPO.
4. Debug training failures using the ladder and error analysis.
5. Estimate compute + memory for any training or inference setup.
6. Build an agent that uses tools to accomplish tasks.

This is the current skill floor for a DL engineer or research student.

What to read next

• Bishop & Bishop · Deep Learning: Foundations and Concepts — for more mathematical rigor.
• Karpathy's Zero to Hero — keep going beyond what we covered.
• Prince's UDL — revisit the chapters you skimmed.
• Recent papers — set up arXiv alerts for your interest area.
• Blog posts · Lil'Log (Lilian Weng), Simon Willison, Chip Huyen, Anthropic engineering blog.

What to do next

• Replicate a paper · picking one from NeurIPS / ICML / ICLR 2025 and implementing it end-to-end teaches you more than any course.
• Contribute to open source · HuggingFace, vLLM, PyTorch. Start with issues labeled "good first issue."
• Build a small project · fine-tune an LLM on your domain; train a tiny diffusion model on a toy dataset; ship an agent.
• Read safety & alignment work · if you care about where this technology is going.