A 1.4x speedup over well-tuned AdamW at 1B+ parameters
Difficulty 50, the highest in the pack: fair-tuning studies show matrix optimizers' gains shrinking from 1.4x to 1.1x between 0.1B and 1.2B, and the best 2026 follow-up reaches 1.2-1.3x, so the target needs a trend reversal; comparable to the feature-learning scaling-exponents problem (47). Application value 70: the optimizer is in every pre-training run. Leverage 42: it would reshape the default training recipe.
The problem
Does any optimizer train a 1.2-billion-parameter language model 1.4 times faster than a carefully tuned AdamW?
- 1Papers claim 1.4x to 2x
- 2Fair tuning shrinks the gain
- 3About 1.1x at 1.2B parameters
- 4Find a method that keeps 1.4x
What is asked. Find an optimizer that reaches a fixed validation loss in at most of the training steps that a fully tuned AdamW needs, for language models of at least 1.2 billion parameters, with both optimizers' settings tuned on equal budgets. An optimizer is the rule that turns the derivatives of the loss into weight updates; AdamW, which gives each weight its own adaptive step size and shrinks weights slightly each step (weight decay), is the default for large models.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Adam, AdamW and the optimizer zooAlgorithm · host
- Then
- A 1.4x speedup over well-tuned AdamW at 1B+ parametersThe problem
State of the art
Wen et al. (preprint) studied ten optimizers across four model scales (0.1B to 1.2B) and data-to-model ratios of 1x to 8x Chinchilla, and report that many claimed 1.4x to 2x speedups shrink under fair tuning: the fastest, matrix-based optimizers such as Muon and Soap, go from about 1.4x at 0.1B parameters to about 1.1x at 1.2B parameters. Liu et al. (preprint) report roughly 2x compute efficiency for Muon under compute-optimal training in their scaling experiments and train a 3B-active / 16B-total MoE model (Moonlight) on 5.7T tokens. The two reports measure against different AdamW baselines, so the gap between them is itself unresolved. A follow-up by Wen et al. (Hyperball, arXiv:2606.16899, preprint) fixes the norms of weights and updates and reports a 20–30% token-equivalent speedup for Muon over weight-decay baselines on models up to 1.2B parameters, still short of 1.4x.
Realistic applications
A 1.4x step-count saving at 1B+ parameters would reduce the cost of every pre-training run by a comparable factor, since the optimizer is the one component used by all of them.
What counts as solved
A public, tuned comparison at at least 1.2B parameters (single runs allowed if the loss gap is far larger than the seed spread measured at 150M), showing steps-to-target ratio of at least 1.4 against AdamW with both sides tuned over matched budgets.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Dataset: a fixed public corpus (for example the one used by the Wen et al. study) with identical data order for both optimizers. Metric: steps (or tokens) to reach a target validation loss that AdamW reaches at the end of its schedule. Threshold: ratio of at least 1.4. Budget: hyperparameter sweeps at 150M parameters with 3 seeds for variance, then at least a 1.2B model at 1x, 2x and 4x Chinchilla tokens, each arm tuned over an equal number of runs (a budget of the order of 10^21 FLOPs per arm). Report weight decay, warm-up and schedule for both.
Claims under verification
+ add a claimWhere the work is
A new optimizer idea is needed and then expensive validation at scale.
Bottlenecks
ranked by votes- #1TheoryGains shrinking with scale
Existing evidence is that preconditioner gains fall from about 1.4x to about 1.1x between 0.1B and 1.2B parameters.
Community grades
Sign in to add your own grade.
Connected problems
Progress
- Sep 2, 2025Fair-tuning study (preprint)source
Wen et al. report speedups over tuned AdamW shrink to about 1.1x at 1.2B parameters.