Interactive Explainer

Neural Processes, Live

A neural net that learns "priors over functions" the way a Gaussian process does — but amortised. Click context points; a Conditional Neural Process predicts the rest with uncertainty that bows out where data is missing. Compare to a real GP fit on the same context.

~20 min Probability · Meta-Learning · Uncertainty

Prelude

What problem does an NP solve?

You have a small context set of $(x, y)$ pairs and you want to predict $y$ at new $x$ values, with uncertainty. A Gaussian process does this with a kernel. A neural process replaces the kernel with a learned encoder/decoder, then amortises inference: at test time it's a single forward pass — no optimisation, no kernel-matrix inversion.

GP. Beautiful uncertainty, but $O(N^3)$ in the context size and you have to pick a kernel.
NP. Cheap to evaluate, no kernel choice. But you must train it on a distribution of related tasks first.

The slogan. An NP is a meta-learned surrogate for a GP. You pay a one-time training cost over many functions; thereafter every prediction is one forward pass.

Step 1

Architecture: encoder → aggregate → decoder

The Conditional Neural Process (Garnelo et al., 2018):

$h$ is a small MLP that turns each context pair $(x_i, y_i)$ into a representation $r_i$. The aggregator is a permutation-invariant pooling (mean) into a single representation $r$. The decoder $g$ maps $(x_*, r)$ to the predicted mean and variance at any query $x_*$.

The CNP pipeline. Mean-pooling makes the model permutation-invariant in the context set.

Step 2

Try it: click points, see the posterior

Click anywhere on the canvas to add a context point. Right-click (or shift-click) to remove. The model is pre-trained on a family of smooth functions; it's not training as you click — it just runs a single forward pass per point. The GP fit on the same context is shown for comparison.

variant |context| = 0

Neural Process posterior (mean ± 2σ)

GP posterior on same context (RBF)

What to watch for. With 0 context points, the NP returns the prior mean of its training family (often roughly zero) with a wide band; the GP returns its prior. Add a few points and both shrink the band locally. The NP's band has a slightly different shape away from the data — the NP has learned what kinds of functions are likely (smoothness, periodicity) from the family it was trained on; the GP only knows its kernel hyperparameters.

Step 2½

The training objective — meta-learning, formally

NPs are trained by meta-learning: simulate many small datasets and ask the NP to predict held-out points within each. For one task with context $\mathcal{C} = \{(x_i, y_i)\}$ and target $\mathcal{T} = \{(x_j, y_j)\}$ the deterministic CNP minimises

where $\mu, \sigma^2$ come from the decoder conditioned on the aggregated representation $r_\mathcal{C}$. The expectation is over many sampled $(\mathcal{C}, \mathcal{T})$ pairs from a task distribution.

The Latent NP introduces a global latent $z$ and uses an ELBO:

$q(z|\mathcal{C} \cup \mathcal{T})$ is the variational posterior (encoder over context + target); $p(z|\mathcal{C})$ is the prior (encoder over context only). Sampling $z$ gives function-consistent samples across queries — drawing whole functions, like a GP. The CNP is the degenerate case where $z$ collapses to a point estimate.

Step 3

The variants and what each adds

CNP. Deterministic. Predicts mean + variance per query. Cheap, single forward pass. Cannot sample whole consistent functions.
Latent NP (LNP). Adds a stochastic latent $z$ that summarises the function. Sample $z$ once, decode many queries — you get a coherent function sample, like a GP draw. Pays for it in a tighter ELBO loss and harder training.
Attentive NP (ANP). Replaces mean aggregation with attention from query to context. The kernel-like behaviour of GPs reappears: each query looks up the most relevant context points. Best of both worlds for many tasks.
Convolutional NP (ConvNP). Uses a convolutional encoder when the input is a 1-D signal / time-series — respects translation equivariance. Strong on sensor / climate data.
Transformer NP / NP with attention only (NeuralProcessFamily). Modern variants tend to drop the explicit aggregator and use a transformer over (context ⊕ query) tokens.

Step 4

When to reach for an NP vs a GP

Many similar tasks, each with little data. Per-sensor calibration, per-patient personalisation, per-image few-shot regression. NPs shine: train once across the family; predict on the next instance in milliseconds.
One task with a lot of data. A GP with inducing points (SVGP) or an exact GP if $N \le 10^4$ usually beats an NP. The NP's amortisation premium disappears.
Online streaming. NPs handle "context grows over time" naturally because each prediction is a forward pass. GPs need to refit (or use online updates).
Image / 2-D completion. Conditional-NP variants do CelebA image completion with one forward pass from $k$ pixels to the full image — a use case where the meta-training distribution over images is natural.
Bayesian optimisation. Transformer Neural Processes (TNP) are competitive surrogates for BO, with the same kind of amortisation: train once on a family of objective functions, deploy without per-objective hyperparameter fitting.

Final takeaway. Neural processes are GPs amortised. If you have many small related tasks and want fast predictions with calibrated uncertainty, they are the right tool. If you have a single big task and want crisp probabilistic semantics, a GP (or deep kernel learning) still wins.

Step 5

Failure modes and how to spot them

Underconfidence. CNPs trained with Gaussian likelihood often produce too-wide intervals — the model hedges against the meta-train distribution. Diagnose with reliability diagrams; fix with attention (ANP), more expressive likelihoods, or post-hoc calibration.
Posterior collapse in LNPs. If the decoder is too strong, the latent $z$ becomes uninformative and the LNP degenerates to a CNP. Fix with $\beta$-VAE-style annealing or free-bits.
Out-of-distribution context. If the test context comes from a function family not seen in meta-training, the NP extrapolates badly. A real GP would just widen its bands; the NP can confidently make up a wrong answer.
Permutation invariance break. Make sure the aggregator (mean / attention) is truly permutation-invariant. A subtle bug to test for: shuffle context order and check the prediction is bit-identical.

Step 6

Reading list

Garnelo et al. (2018) — Conditional Neural Processes.
Garnelo et al. (2018) — Neural Processes. (Latent-variable variant.)
Kim et al. (2019) — Attentive Neural Processes.
Gordon et al. (2020) — Convolutional Conditional Neural Processes.
Nguyen & Grover (2022) — Transformer Neural Processes.
Bruinsma et al. (2023) — The Neural Process Family textbook chapter; the standard pedagogical reference.