Mixture of ExpertsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Mixture of Experts

Scaling laws (previous lesson) say bigger models are better, but bigger models cost more per token, for training and every inference. Mixture of Experts (MoE) breaks that link: it grows a model's parameter count enormously while keeping the compute per token nearly flat, by activating only a small sparse slice of the network for each input. It's how many of the largest frontier models pack hundreds of billions or trillions of parameters into a serveable budget.

Real-world examples: the Switch Transformer, GLaM, Mixtral, and a large fraction of frontier LLMs use MoE feed-forward layers; it's the dominant technique for scaling parameters past what a dense model can afford.

The idea: replace one FFN with many experts

In a standard transformer block, every token passes through the same feed-forward network (FFN). An MoE layer replaces that single FFN with NN parallel experts (each its own FFN) plus a small router (gating network). For each token, the router picks a few experts; the token is processed by only those.

dense: every token→1 big FFNMoE: every token→top-k of N experts\text{dense: every token} \to \text{1 big FFN} \qquad \text{MoE: every token} \to \text{top-}k\text{ of }N\text{ experts}

The viz shows the routing: each token lights up only its chosen experts.

Mixture-of-Experts routing

tokensThecatsatonthematexperts (load)E03E12E22E32E43
experts5
active / token2 of 5
selected"cat" → E3, E0

A gating network scores each token against every expert; the token is processed by only its top-k experts (sparse routing), and their outputs are combined by the gate weights. Pick a token to see its experts and gate weights. The per-expert load is what load-balancing losses keep even — an overloaded expert bottlenecks, an idle one wastes capacity.

Top-k gating

The router is a learned linear layer producing a score per expert; a softmax over the top-kk chosen experts gives the combination weights. The layer's output is the gated sum of just those experts:

g=softmax(top-k( Wr x )),y=∑e∈top-kge Experte(x)g = \text{softmax}\big(\text{top-}k(\,W_r\,\mathbf{x}\,)\big), \qquad \mathbf{y} = \sum_{e \in \text{top-}k} g_e \, \text{Expert}_e(\mathbf{x})

Typically k=1k = 1 (Switch Transformer) or k=2k = 2 (most others) out of N=8,64,N = 8, 64, or more experts. Because only kk of NN experts run, the FLOPs per token depend on kk, not NN.

The headline: parameters ≫ compute

This is the whole point. With NN experts but only kk active:

total params∝N(huge),compute per token∝k(small, fixed)\text{total params} \propto N \quad\text{(huge)}, \qquad \text{compute per token} \propto k \quad\text{(small, fixed)}

A model with 64 experts has ~64× the FFN parameters of a dense model but, at k=1k=1, roughly the same compute per token. More parameters store more knowledge; sparse activation keeps each token cheap. MoE decouples capacity from cost — the lever scaling laws couldn't otherwise reach affordably.

Concrete numbers — Mixtral 8×7B: N=8N = 8 experts per layer, k=2k = 2 active. Total parameters ≈ 46.7 B, but each token activates only ≈ 12.9 B of them, so it stores knowledge like a ~47 B model while running (per token) like a ~13 B model.

Specialization is emergent, not assigned

You don't tell experts what to specialize in — the router learns to send related tokens to the same experts. In practice experts often loosely specialize (by syntax, language, or topic), though the structure is fuzzy and far from human-interpretable. The router and experts are trained jointly, end to end.

Load balancing: the central difficulty

Left alone, the router collapses: it learns to send almost everything to a few favorite experts, leaving the rest idle. That wastes the extra capacity and creates a compute bottleneck on the popular experts. Two standard fixes:

  • Auxiliary load-balancing loss: an added penalty that encourages the router to spread tokens evenly across experts (e.g. penalizing the product of each expert's token fraction and its average gate probability).
  • Expert capacity: a hard cap on how many tokens each expert will accept per batch; tokens beyond the cap are dropped (skipped via the residual connection) to keep the computation balanced and hardware-friendly.

MoE's real costs

The parameter savings aren't free. MoE adds: memory (all NN experts' weights must be stored/loaded even though few run), communication (experts are sharded across devices, so routing shuffles tokens between GPUs — "all-to-all"), and training instability (the discrete top-k routing is finicky). MoE wins on compute per token, not on memory or simplicity.

When MoE helps

  • ✅ Knowledge-heavy tasks at scale where you want maximum parameters per unit of inference compute.
  • ✅ Settings where memory and inter-device bandwidth are available.
  • ❌ Memory-constrained deployment (all experts must be resident) or small models where a dense FFN is simpler and the routing overhead isn't worth it.

Exercise

In a Mixture-of-Experts layer with N experts and top-k routing, how many experts process each token?

Need a hint?

What does 'sparse' activation mean here?

Exercise

Why is a Mixture-of-Experts model said to 'decouple parameters from compute'?

Need a hint?

Which quantity scales with N, and which with k?

Exercise

Why do MoE models need an auxiliary load-balancing loss (and/or expert capacity limits)?

Need a hint?

What does the router do if left unconstrained?

Common mistakes

  1. Equating params with compute. An MoE's giant parameter count does not mean proportionally more FLOPs per token — only kk experts run.
  2. Ignoring load balancing. Without an auxiliary loss / capacity limit, routing collapses onto a few experts and the rest are dead weight.
  3. Forgetting the memory bill. Every expert's weights are resident even if rarely used — MoE spends extra memory to save compute per token, not the other way around.
  4. Assuming clean expert specialization. Specialization is emergent and fuzzy, not a designed, interpretable decomposition.

Key takeaways

  • MoE replaces a block's single FFN with NN experts + a router; each token uses only its top-kk experts (sparse activation).
  • This decouples parameters from compute: total params scale with NN (huge) while FLOPs per token scale with kk (small, fixed) — more capacity at near-constant cost.
  • Top-kk gating (usually k=1k=1 or 22) selects and weights experts via a learned softmax router.
  • Load balancing (auxiliary loss + expert capacity) is essential to stop routing collapse; MoE's costs are memory, cross-device communication, and training instability.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Mixture of ExpertsRecurrent Neural NetworksCalculus for MLFoundation Models & Scali…The Transformer Architect…GPUs for Deep Learning