Mixture of Experts
MoE routes each token (or example) to a sparse subset of specialist feed-forward experts, raising parameter count without paying dense FLOPs on every token.
Definition
A router produces scores over N experts; top-k experts fire. Shazeer et al. (2017) made this work at Google scale; Switch Transformer uses k=1. Load-balancing losses stop all traffic collapsing onto one expert.
Why it matters
Sparse MoE is how some of the largest LMs grow parameters faster than compute. Routing quality is the whole product: a bad router is a random ensemble that wastes capacity.
Case
Switch Transformer showed that replacing every other FFN with a 1-expert-per-token switch can match dense quality at smaller FLOP budgets — if you keep experts balanced.
How to read it
Report expert utilization and dropped-token rates, not only perplexity. An MoE with one expert doing 90% of the work is a dense model with extra RAM.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer et al. · 2017 · ICLRSparse MoE at scale.
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityFedus, Zoph, Shazeer · 2022 · JMLRk=1 routing.