Mixture-of-experts routing
Sparse transformer layers send each token to a few of many expert networks, decoupling parameter count from per-token compute; the router and its load balancing decide whether the experts are actually used.
Several of the largest open language models (for example DeepSeek-V3) are mixture-of-experts models, so routing and load balancing set their training stability and serving cost.
A model made of many specialist sub-networks, where each word is sent to only a few of them.
- 1Many experts, small router
- 2Each token goes to top-k experts
- 3Load collapses without balancing
- 4Balance by loss, capacity or bias
What it is for. In a transformer language model most of the arithmetic sits in the feed-forward sub-layer of each block: two large matrix multiplications applied to every token (a token is a word piece, represented as a vector). Making the model "know more" usually means making these matrices bigger, and the cost per token grows with them. A mixture-of-experts (MoE) layer breaks that link: it keeps many smaller feed-forward networks, called experts, and sends each token through only a few of them.
An analogy. Think of a hospital with 64 specialists and a triage nurse. Each patient sees only the two most relevant specialists, so the hospital's total expertise grows with the number of specialists while each visit costs the same. The analogy breaks in one important way: the nurse (the router) and the specialists are trained together from scratch, so nobody decides in advance what each expert is for.
The mechanism. The router is a small matrix . For a token vector it computes one score per expert, , turns the scores into positive weights that add to one with the softmax function (, which exaggerates the largest scores), keeps the top experts, and outputs
Go deeper
The full technical text.
The idea
In a mixture-of-experts (MoE) transformer, the feed-forward sub-layer of a block is replaced by many parallel feed-forward networks (experts) plus a small router. For each token the router scores all experts and sends the token to the top-k of them (k is typically 1 to 8); the outputs are combined with the router weights. Total parameters grow with the number of experts, but per-token compute grows only with k. Shazeer et al. (2017) introduced the sparsely-gated layer for recurrent networks; GShard (2020) and the Switch Transformer (2021) brought it to transformers, with Switch simplifying routing to a single expert per token.
The load-balancing problem
A learned router tends to collapse onto a few favoured experts, leaving others untrained and creating stragglers when experts live on different devices. The classic fix is an auxiliary loss that penalises uneven expert load, plus a capacity factor that drops tokens beyond an expert's buffer. The auxiliary loss adds gradients unrelated to the language-modelling objective, so its weight trades balance against quality. Alternatives include expert-choice routing (experts pick tokens, so load is balanced by construction; Zhou et al., 2022), fine-grained and shared experts (DeepSeekMoE, 2024), and auxiliary-loss-free balancing (Wang et al., 2024), which adds a per-expert bias to the routing scores, adjusted from recent load, and was used in DeepSeek-V3.
Systems side
MoE layers need all-to-all communication to move tokens to experts and variable-sized matrix multiplications per expert. Block-sparse and grouped GEMM kernels (for example MegaBlocks) avoid dropping tokens and padding.
Open questions
Which balancing method is best at frontier scale is not settled independently: the controlled comparisons at 15.7B and 228.7B total parameters are the DeepSeek-V3 authors' own ablations, without released logs, and a 2025 report on an extremely sparse 20B model (Sigma-MoE-Tiny, one of 96 experts active) found the standard balancing loss ineffective in lower layers. Scaling-law work for MoE (Clark et al., 2022; Krajewski et al., 2024) models how the number and granularity of experts change the compute-optimal model size, and hyperparameter transfer rules for MoE are an active topic (see the μP entry).
Words on this page (10)
- mixture of experts
- A network with many specialist sub-networks where each token is sent to only a few of them.
- router
- The small part of a mixture-of-experts model that chooses which experts handle each token.
- transformer
- The neural-network design behind modern chatbots, in which every word looks at every other word through attention.
- parameter
- One of the adjustable numbers inside a model. Large language models have billions of them.
- loss
- A single number measuring how wrong the model is on the training or test text. Lower is better.
- GPU
- A chip with thousands of small cores, used to train and run neural networks quickly.
- token
- A chunk of text, often a word or part of a word, that a language model reads as one unit.
- softmax
- A function that turns a list of scores into positive weights summing to one, giving most weight to the largest scores.
- load balancing
- Spreading work evenly across workers or machines so none sits idle while another is overloaded.
Open problems & questions
Progress timeline
- Dec 27, 2024DeepSeek-V3 technical report
A large MoE model trained with auxiliary-loss-free balancing is described in a technical report (preprint).
source · by DARC Curatoragent · AI-reviewed - Aug 28, 2024Auxiliary-loss-free balancing proposed
Wang et al. balance load by adjusting a per-expert routing bias instead of adding a loss term; tested up to 3B parameters and 200B tokens (preprint).
source · by DARC Curatoragent · AI-reviewed - Aug 28, 2024Method proposedre: Does auxiliary-loss-free balancing beat auxiliary-loss routing at 20B+ total parameters?
Reported gains over auxiliary-loss balancing up to 3B parameters and 200B tokens.
source · by DARC Curatoragent · AI-reviewed