Interactive Explainer

Mixture of Experts, Live

Four linear experts and a softmax gate, trained jointly in your browser on a piecewise 2-D regression. Watch the gate carve the input plane into four regions, the experts specialise inside their regions, and the load-balancing auxiliary loss wake up dead experts. Toggle soft mixture vs top-1 hard routing.

~18 min Deep Learning · Sparse Models · Routing

Prelude

Why MoE

A single dense network has to learn one function for every input. As models grow, more parameters means more compute on every forward pass — the cost of capacity scales linearly with capacity itself.

Mixture of Experts decouples them. You have $E$ expert networks and a small gating network. For each input, the gate picks one or two experts; only those run. A model can have 8× the parameters of a dense counterpart but cost roughly 1× the compute — provided routing is sparse.

The slogan. MoE pays for parameters in memory, not in flops. As of 2026 every frontier-scale model (GPT-OSS, Mixtral, DeepSeek-V3, Llama-MoE) is some flavour of MoE.

Step 1

The model in one line

Each expert $f_e$ is a function (we use linear: $f_e(x) = w_e^\top x + b_e$). The gate $g(x) \in \Delta^E$ is a softmax over $E$ logits. The model output is a weighted mixture; the gradient flows into both the experts (where they were used) and the gate (which gets a signal pushing it toward better routings).

Hard top-1 routing replaces the soft mixture with $\hat e = \arg\max g_e(x)$ — only one expert runs per input. We use a straight-through gradient so the gate still learns.

Step 2

Train it live

The toy task: a 2-D plane carved into four quadrants, each with a different linear truth function. A single linear model can't fit it; four linear experts plus a gate can.

routing load-balance loss balance λ 0.10 step 0 MSE —

Gating regions (one colour per expert)

Per-expert prediction error

Loss curve + expert utilisation

What to watch for. With soft mixture + no balance loss the gate sometimes collapses onto a single expert (dead expert problem). Turn on load-balance and the balance term penalises uneven utilisation; the four-way split usually appears within ~150 steps. Switch to top-1 for the Switch-Transformer-style routing — Voronoi-like region boundaries appear instead of a smooth softmax mixture.

Step 3

The auxiliary load-balance loss

Without intervention the gate has every reason to collapse: the easiest local minimum is "always pick expert 1". The auxiliary loss pushes the gate toward uniform utilisation:

$f_e$ is the fraction of tokens routed to expert $e$ and $P_e$ is the average gate probability for expert $e$ over the batch. The product is minimised when both are uniform ($1/E$). The Switch Transformer paper introduced this exact form and it's what every modern MoE uses.

Step 4

What scales, what hurts, what production looks like

Switch Transformer (Fedus et al., 2021). One expert per token (top-1). Demonstrated 4× speed-ups at constant quality.
Mixtral 8×7B (Mistral, 2023). Top-2 routing across 8 experts; only the experts of the top-2 gates run; effective parameter count ~47B with ~13B active per token.
DeepSeek-MoE (2024). Many small experts (256+) with finer-grained routing. Auxiliary-loss-free balancing via per-expert biases on the gate.
Common pitfalls. Routing instability during early training; experts collapse if the auxiliary loss is too small or absent; ALL-to-ALL communication dominates training time at large scale; "expert capacity" overflow needs token dropping or re-routing.
What MoE does not save you. Inference memory: every expert's weights still need to be in GPU/TPU memory. MoE is FLOP-cheap, RAM-expensive.

Step 5

Where MoE shows up beyond LLMs

Multi-task and multi-domain models. One expert per task domain; the gate learns the routing. Useful when many sensor modalities or task types share a backbone.
Time-series with MoE. Different experts for different temporal patterns (activity recognition vs sleep vs heart-rate). The gate becomes a soft regime classifier.
Multimodal models. Image vs text vs audio expert blocks; gate routes per modality (often hard top-1).
Conditional computation in vision. Different experts for different scales / regions in satellite or medical images.

Final takeaway. MoE is the cheapest way to give a model more "knowledge" per parameter. As long as routing is sparse and balancing is in place, you get capacity without flops. Once you have it, dense scaling looks wasteful.

Step 6

Expert parallelism — the systems story

Past ~1B parameters per expert, you can't fit all experts on one GPU. The standard sharding pattern (Lepikhin et al., 2020):

Expert parallelism. Place different experts on different devices. Each device holds a subset of experts.
All-to-all dispatch. For each batch of tokens, send tokens to the device hosting their chosen expert; run the expert locally; send outputs back. This is two all-to-all collectives per layer.
Expert capacity. Each expert has a fixed slot count per batch (e.g. $1.25 \times \text{tokens}/E$). Overflowing tokens get dropped (zero-output) or re-routed to the second-choice expert. The capacity factor is a knob between speed (low capacity, more drops) and quality (high capacity, more compute).
Pipelining + tensor parallelism. Stack expert parallelism on top of standard 3D parallelism. Each device runs one stage's experts, sharded.

The all-to-all is what makes MoE training systems hard. Frameworks: GShard, Tutel, Megablocks, DeepSpeed-MoE.

Step 7

Frontier MoE comparison

Model	Total params	Active / token	# experts	Top-k	Notable trick
Switch Transformer (2021)	up to 1.6T	~1/E	up to 2048	1	Aux load-balance loss; expert capacity
GLaM (2021)	1.2T	97B	64	2	Routing balance via aux loss; sparsity at training
Mixtral 8×7B (2023)	47B	13B	8	2	Open-weight; outperforms Llama-2 70B
DeepSeek-V3 (2024)	671B	37B	256 + 1 shared	8	Aux-loss-free balancing via per-expert bias
Mixtral 8×22B (2024)	141B	39B	8	2	Same architecture, scaled experts
Qwen2-MoE A14B (2024)	57B	14B	64 + 4 shared	4	Shared experts always active

Step 8

Reading list

Jacobs, Jordan, Nowlan, Hinton (1991) — Adaptive Mixtures of Local Experts. The original MoE paper.
Shazeer et al. (2017) — Outrageously Large Neural Networks. Top-k gating, MoE inside Transformers.
Lepikhin et al. (2020) — GShard. The expert-parallelism systems paper.
Fedus, Zoph, Shazeer (2021) — Switch Transformer.
Du et al. (2022) — GLaM.
DeepSeek-AI (2024) — DeepSeek-V3 Technical Report. Aux-loss-free MoE.
Mistral AI (2024) — Mixtral of Experts.