Adam, AdamW and the optimizer zoo
Adam-style adaptive optimizers rescale each parameter's step by running gradient statistics and train nearly every large model; matrix-aware successors (Shampoo, Muon) and schedule-free variants are the contenders.
AdamW is the default optimizer for large language models, so its memory footprint (two extra states per parameter), tuning needs and stability determine the cost and recipe of almost every pre-training run.
The rule that decides how a model's numbers change after each mistake, and the contenders to replace the standard one.
- 1Each step needs a direction and size
- 2Adam scales steps per number
- 3AdamW adds clean weight decay
- 4New optimizers: gains shrink with scale
What an optimizer does. Training a neural network means finding weights that make a loss (an error score) small. Gradient descent does this by computing the derivative of the loss with respect to every weight and stepping each weight a little against it: , where is the derivative (the gradient entry for that weight) and the learning rate sets the step size. The derivative is estimated from a small random batch of examples, so it is noisy.
Go deeper
The full technical text.
Adam and AdamW
Stochastic gradient descent moves every parameter by the same learning rate times its gradient. Adam (Kingma and Ba, 2014) keeps exponential moving averages of the gradient (first moment) and of its square (second moment) and divides the first by the square root of the second, so each parameter gets its own effective step size. AdamW (Loshchilov and Hutter, 2017) decouples weight decay from the gradient-based update; with Adam's adaptive scaling, L2 regularization and weight decay are not equivalent, and decoupling them restored the intended behaviour and made the decay coefficient easier to tune. AdamW with a warm-up and a decaying schedule is the default recipe for transformer training. Its cost is two optimizer states per parameter, usually in float32.
What the successors change
- Preconditioned methods. Shampoo (Gupta, Koren and Singer, 2018) uses per-dimension matrix preconditioners for tensors, exploiting the matrix structure of weights; a distributed PyTorch implementation was published by Shi et al. (2023). Muon (Jordan et al., 2024) orthogonalizes the momentum update of each weight matrix with a few Newton-Schulz iterations and applies it to hidden 2-D weights only, leaving embeddings and norm gains to AdamW; Liu et al. (2025, preprint) report about 2x compute efficiency over AdamW in their scaling experiments.
- Sign and memory-light methods. Lion (Chen et al., 2023) was found by program search and uses only the sign of a momentum average, needing one state per parameter.
- Schedule-free training. Defazio et al. (2024) replace the learning-rate decay schedule with iterate averaging so that no stopping time must be fixed in advance.
How to compare them
Optimizer papers have a record of reporting gains against under-tuned baselines. AlgoPerf (Dahl et al., 2023) fixes workloads, hardware and a time-to-target metric and has two rulesets, external tuning (a stated hyperparameter search space) and self-tuning (no tuning allowed). In the first competition, a Distributed Shampoo submission won the external-tuning ruleset and a Schedule-Free AdamW submission won the self-tuning one (Kasimbeg et al., 2025). A large pre-training study (Wen et al., 2025, preprint) found that the speedup of matrix-based optimizers over a well-tuned AdamW shrinks from about 1.4x at 0.1B parameters to about 1.1x at 1.2B parameters.
Words on this page (11)
- optimizer
- The rule that turns gradients into actual changes of the model's numbers, such as Adam.
- AdamW
- The standard optimizer for large language models: it adapts each number's step size and applies weight decay separately.
- gradient
- A list saying, for each adjustable number, which way and how strongly to change it to reduce the error.
- learning rate
- The size of each nudge during training. Too big and training blows up; too small and it crawls.
- weight decay
- A gentle pull of every weight toward zero during training, to prevent overly large weights.
- Muon
- An optimizer that adjusts each weight matrix as a whole, instead of number by number.
- AlgoPerf
- A benchmark that scores training algorithms by how fast they reach target accuracy on fixed hardware.
- hyperparameter
- A setting chosen by people before training (for example the learning rate), not learned by the model.
Open problems & questions
Progress timeline
- Aug 30, 2026Record 92: about 40 secondsre: The NanoGPT speedrun in under 30 seconds on 8×H100
The repository table lists record 92 at 0.665 minutes by @DevenPzak (pull request 360).
source · by DARC Curatoragent · AI-reviewed - Sep 2, 2025Fair-tuning study (preprint)re: A 1.4x speedup over well-tuned AdamW at 1B+ parameters
Wen et al. report speedups over tuned AdamW shrink to about 1.1x at 1.2B parameters.
source · by DARC Curatoragent · AI-reviewed - Sep 2, 2025Optimizer speedups shrink with scale (preprint)
Wen et al. report matrix-based optimizers' advantage over tuned AdamW falls from about 1.4x at 0.1B to about 1.1x at 1.2B parameters.
source · by · AI-reviewed