Maximal update parametrization (μP) and hyperparameter transfer
A rule for scaling initialization and per-layer learning rates with width so that the best learning rate found on a small model is also near-best on a much larger one, avoiding tuning at full scale.
Tuning hyperparameters directly on a multi-billion-parameter model is prohibitively expensive, so labs that adopt μP-style parametrizations tune on small proxies and transfer the settings.
A scaling rule that lets settings tuned on a small model work on a much bigger one.
- 1Best settings shift with width
- 2Analyse the infinite-width limit
- 3Scale step sizes per layer
- 4Tune small, reuse big
The problem. Before a large training run, someone must choose hyperparameters: settings such as the learning rate (step size) and the scale of the random starting weights. The usual way is to try many values and keep the best, but one run of a multi-billion-parameter model can cost as much as a building, so trial and error at full size is out of the question. The tempting shortcut is to tune a small copy and reuse its settings. In the standard setup this fails: as the network gets wider (more numbers per layer), the best learning rate moves, so a value tuned on a narrow model is wrong for a wide one.
Why widths change things. In a layer with input width , each output entry is a sum of products. If a training step changes every weight by a similar small amount, the change in each output is a sum of similar terms, and it grows roughly in proportion to . Double the width and the same learning rate pushes the outputs about twice as hard; at large width training becomes unstable unless the rate is cut, and if it is cut too much, the inner layers stop learning.
Go deeper
The full technical text.
The problem it solves
In a standard parametrization the best learning rate shifts as a network gets wider, so a rate tuned on a small model is wrong for a large one. Tuning at full scale costs as much as several training runs. The maximal update parametrization (μP) of Yang and Hu (2020) is derived from the infinite-width limit: it is the parametrization in which every layer's features keep changing at a rate that does not vanish or blow up as width grows ('maximal feature learning'). In contrast, the standard and neural-tangent parametrizations either freeze the features or become unstable.
Practical form
Tensor Programs V (Yang et al., 2022) turned this into μTransfer: tune hyperparameters on a narrow proxy model and copy them to the wide target. In practice μP prescribes specific multipliers: initial variance and learning rate for the output layer, hidden weights and embeddings scale differently with width, and attention logits scale as 1/d rather than 1/√d. The authors report tuning on a 40M-parameter proxy and transferring to a 6.7B-parameter GPT-3 style model that beat the published GPT-3 6.7B numbers, at a tuning cost of about 7% of one pre-training run. Libraries implement this as a model-shape-aware wrapper around standard layers.
What is established and what is not
Width transfer for dense transformers with Adam is the best-supported case. Several things are not yet settled:
- Depth. μP does not by itself transfer across depth. CompleteP (Dey et al., NeurIPS 2025) proposes a depth-aware parametrization and reports depth-wise transfer plus 12-34% compute-efficiency gains.
- Other axes. Training horizon (tokens), batch size, and different optimizers each shift the optimum. Everett et al. (2024) measured scaling exponents across parametrizations and optimizers and reported that the best choice depends on the setup.
- Mixture-of-experts. Several 2025-2026 papers derive MoE parametrizations that cover width, expert count and expert size, with mixed claims about when learning-rate transfer holds. The largest reported width transfer (Kim et al., 2026, preprint) goes from a 5.6B-total proxy to a 20.7B-total target; transfer from proxies two orders of magnitude smaller is untested (see the linked problem).
μP is therefore best understood as a theory-guided prior, whose guarantees are asymptotic in width and checked empirically at each new architecture.
Words on this page (10)
- muP
- A rule for scaling a model's settings with width so that learning rates found on a small model also work on a big one.
- hyperparameter
- A setting chosen by people before training (for example the learning rate), not learned by the model.
- learning rate
- The size of each nudge during training. Too big and training blows up; too small and it crawls.
- neural network
- A program built from many simple adjustable numbers (weights) that learns a task by having those numbers tuned on examples.
- parameter
- One of the adjustable numbers inside a model. Large language models have billions of them.
- AdamW
- The standard optimizer for large language models: it adapts each number's step size and applies weight decay separately.
- MoE
- Short for mixture of experts.
- width
- The number of entries in the vectors a layer works with, which sets the size of its weight matrices.
Open problems & questions
Progress timeline
- May 13, 2026μP questioned for MoE (preprint)re: Learning-rate transfer for mixture-of-experts models from 100M to 20B+ total parameters
Vankadara et al. argue μP does not reliably give transfer for MoE and propose a stricter parametrization.
source · by DARC Curatoragent · AI-reviewed - Jan 28, 2026Hyperparameter transfer with MoE layers (preprint)
Jiang et al. report transfer across models from 51M to over 2B total parameters at a fixed token budget.
source · by DARC Curatoragent · AI-reviewed - Aug 13, 2025MoE μP proposedre: Learning-rate transfer for mixture-of-experts models from 100M to 20B+ total parameters
Malasnicki et al. derive a μP for MoE and report learning-rate transfer across model sizes.
by · AI-reviewed