Mixture of Experts
Scaling laws (previous lesson) say bigger models are better, but bigger models cost more per token, for training and every inference. Mixture of Experts (MoE) breaks that link: it grows a model's parameter count enormously while keeping the compute per token nearly flat, by activating only a small sparse slice of the network for each input. It's how many of the largest frontier models pack hundreds of billions or trillions of parameters into a serveable budget.
Real-world examples: the Switch Transformer, GLaM, Mixtral, and a large fraction of frontier LLMs use MoE feed-forward layers; it's the dominant technique for scaling parameters past what a dense model can afford.
The idea: replace one FFN with many experts
In a standard transformer block, every token passes through the same feed-forward network (FFN). An MoE layer replaces that single FFN with parallel experts (each its own FFN) plus a small router (gating network). For each token, the router picks a few experts; the token is processed by only those.
The viz shows the routing: each token lights up only its chosen experts.
Mixture-of-Experts routing
A gating network scores each token against every expert; the token is processed by only its top-k experts (sparse routing), and their outputs are combined by the gate weights. Pick a token to see its experts and gate weights. The per-expert load is what load-balancing losses keep even — an overloaded expert bottlenecks, an idle one wastes capacity.
Top-k gating
The router is a learned linear layer producing a score per expert; a softmax over the top- chosen experts gives the combination weights. The layer's output is the gated sum of just those experts:
Typically (Switch Transformer) or (most others) out of or more experts. Because only of experts run, the FLOPs per token depend on , not .
The headline: parameters ≫ compute
This is the whole point. With experts but only active:
A model with 64 experts has ~64× the FFN parameters of a dense model but, at , roughly the same compute per token. More parameters store more knowledge; sparse activation keeps each token cheap. MoE decouples capacity from cost — the lever scaling laws couldn't otherwise reach affordably.
Concrete numbers — Mixtral 8×7B: experts per layer, active. Total parameters ≈ 46.7 B, but each token activates only ≈ 12.9 B of them, so it stores knowledge like a ~47 B model while running (per token) like a ~13 B model.
Specialization is emergent, not assigned
You don't tell experts what to specialize in — the router learns to send related tokens to the same experts. In practice experts often loosely specialize (by syntax, language, or topic), though the structure is fuzzy and far from human-interpretable. The router and experts are trained jointly, end to end.
Load balancing: the central difficulty
Left alone, the router collapses: it learns to send almost everything to a few favorite experts, leaving the rest idle. That wastes the extra capacity and creates a compute bottleneck on the popular experts. Two standard fixes:
- Auxiliary load-balancing loss: an added penalty that encourages the router to spread tokens evenly across experts (e.g. penalizing the product of each expert's token fraction and its average gate probability).
- Expert capacity: a hard cap on how many tokens each expert will accept per batch; tokens beyond the cap are dropped (skipped via the residual connection) to keep the computation balanced and hardware-friendly.
MoE's real costs
The parameter savings aren't free. MoE adds: memory (all experts' weights must be stored/loaded even though few run), communication (experts are sharded across devices, so routing shuffles tokens between GPUs — "all-to-all"), and training instability (the discrete top-k routing is finicky). MoE wins on compute per token, not on memory or simplicity.
When MoE helps
- ✅ Knowledge-heavy tasks at scale where you want maximum parameters per unit of inference compute.
- ✅ Settings where memory and inter-device bandwidth are available.
- ❌ Memory-constrained deployment (all experts must be resident) or small models where a dense FFN is simpler and the routing overhead isn't worth it.
Exercise
In a Mixture-of-Experts layer with N experts and top-k routing, how many experts process each token?
Need a hint?
What does 'sparse' activation mean here?
Exercise
Why is a Mixture-of-Experts model said to 'decouple parameters from compute'?
Need a hint?
Which quantity scales with N, and which with k?
Exercise
Why do MoE models need an auxiliary load-balancing loss (and/or expert capacity limits)?
Need a hint?
What does the router do if left unconstrained?
Common mistakes
- Equating params with compute. An MoE's giant parameter count does not mean proportionally more FLOPs per token — only experts run.
- Ignoring load balancing. Without an auxiliary loss / capacity limit, routing collapses onto a few experts and the rest are dead weight.
- Forgetting the memory bill. Every expert's weights are resident even if rarely used — MoE spends extra memory to save compute per token, not the other way around.
- Assuming clean expert specialization. Specialization is emergent and fuzzy, not a designed, interpretable decomposition.
Key takeaways
- MoE replaces a block's single FFN with experts + a router; each token uses only its top- experts (sparse activation).
- This decouples parameters from compute: total params scale with (huge) while FLOPs per token scale with (small, fixed) — more capacity at near-constant cost.
- Top- gating (usually or ) selects and weights experts via a learned softmax router.
- Load balancing (auxiliary loss + expert capacity) is essential to stop routing collapse; MoE's costs are memory, cross-device communication, and training instability.
Related concepts
- Foundation Models & Scaling Laws: why we want more parameters, the demand MoE serves cheaply
- Transformer Architecture: the FFN sublayer MoE replaces
- GPUs for Deep Learning: the memory/communication trade-offs MoE's sharding runs into