In-Context Learning, From the Inside

What ICL is, operationally

You pass a frozen model a prompt like:

apple → red
banana → yellow
broccoli → green
strawberry →

and the model outputs red. No fine-tuning, no gradient steps. The accuracy on the task usually grows with the number of in-context examples $k$, then plateaus. Empirically this works for translation, arithmetic, named entity recognition, function approximation, even small reasoning tasks.

Two facts that motivated the research:

1. ICL emerges with scale. GPT-2 (1.5B) does it poorly; GPT-3 (175B) does it well. Below a critical model size and training-data scale, ICL accuracy is at chance.
2. ICL works on tasks that don't appear in pretraining. The model isn't retrieving memorised answers; it's running something that functions like an algorithm at inference.

The induction head — a 2-layer attention circuit

Olsson et al. (2022) localised the mechanism behind the simplest form of ICL — copying. It needs two attention heads working together:

• Layer 1 — previous-token head. At each position $t$, attend to position $t-1$. Now every token's representation contains "the previous token".
• Layer 2 — induction head. Given the current token $A$ at position $t$, search earlier in the sequence for places where the previous token was also $A$. Found one? Predict whatever followed it.

That two-step circuit implements: "find the last time we saw this token; copy what came next." On a sequence X A Y B A ? it predicts Y. Combined with the attention article's mechanics, this is one short script.

Watch it work — k-shot copy task, live

A toy task: each prompt has $k$ (key, value) pairs followed by a query key. The correct answer is the value of the pair whose key equals the query. The model is a 2-layer attention network with 4 heads pre-trained to do well at copy tasks. Slide $k$; watch accuracy.

"ICL is implicit gradient descent"

A surprising line of work (Akyürek et al. 2023; von Oswald et al. 2022; Garg et al. 2022) shows that a single Transformer layer applied to a sequence of $(x_i, y_i)$ pairs can implement a step of gradient descent on linear regression. With enough layers, ICL approximately equals "fit a small model to the prompt's examples, then predict."

The construction: pack $(x, y)$ pairs into tokens; the attention layer's $W_Q, W_K, W_V$ can be chosen so the output at the query position equals the prediction of one step of GD on the data so far. Layer-by-layer compounds into multiple GD steps. This explains both why ICL needs scale (enough layers to do enough steps) and why it generalises to held-out function classes.

Practical implications

• More examples isn't always better. Accuracy plateaus around $k=10-20$ for most tasks; beyond that you're just using context for diminishing returns.
• Example selection matters. Diverse, representative examples outperform random sampling. The "best-of-K with semantic search over a labelled pool" is the RAG-flavoured improvement.
• Format sensitivity is real. The exact separator, capitalisation, and ordering of examples can swing accuracy by 5-15 points. Test variants.
• Chain-of-thought is ICL with reasoning steps. Including "let's think step by step" plus worked examples unlocks much harder tasks. Why: the model gets to allocate more compute to the answer.
• Calibration breaks badly under ICL. Confidence is unreliable; wrap critical predictions in conformal prediction.
• Fine-tune vs ICL tradeoff. If you have ≥100 labelled examples and care about reliability, LoRA fine-tune; if you have ≤30 examples or constantly-changing tasks, ICL is the right tool.