Neural scaling laws and compute-optimal training
Empirical power laws relating loss to model size, data and compute, used to choose how big a model to train and on how many tokens before spending the budget.
Labs set model size and token count for multi-million-dollar training runs from fitted scaling laws, so errors in the fits translate directly into wasted compute.
Smooth rules predicting how a model's error falls as it gets bigger and sees more text.
- 1Train many small models
- 2Fit a power law to the losses
- 3Extrapolate to the big run
- 4Choose size and data together
What they are for. A large language-model training run costs millions of dollars. Before spending that, a lab must decide how big the model should be (parameters, ) and how much text to train it on (tokens, ). Scaling laws are fitted formulas that predict the final loss from and , using a series of much cheaper small runs.
The loss. The model is trained to predict the next token. Its loss, cross-entropy, measures how surprised it is on average by the true next token: if it gave the right token probability , the surprise is . Lower loss means better prediction.
Go deeper
The full technical text.
What the laws say
Kaplan et al. (2020) found that the cross-entropy loss of transformer language models falls as a smooth power law in non-embedding parameters N, dataset size D and training compute C, over many orders of magnitude, when the other factors are not bottlenecks. Hoffmann et al. (2022, 'Chinchilla') refitted the relationship with a parametric form in which loss is an irreducible term plus a power law in N plus a power law in D, and concluded that parameters and tokens should be scaled in roughly equal proportion for a fixed compute budget. Their 70B-parameter model trained on 1.4T tokens (about 20 tokens per parameter) outperformed a much larger model trained on fewer tokens. Chinchilla gave three estimation approaches (minimum envelope over training curves, IsoFLOP profiles, and a parametric fit); they are meant to agree.
Refinements and disputes
- Reconciling Kaplan and Chinchilla. Pearce and Song (2024) attribute most of the discrepancy to Kaplan counting non-embedding parameters and working at small scale.
- Replication. Besiroglu et al. (2024, preprint) reconstructed the parametric fit from data extracted from the Chinchilla figures and report that the published fit was inconsistent with the other two approaches, and that its confidence intervals were implausibly narrow for the number of experiments run; their own refit agreed with approaches 1 and 2.
- Repeated data. Muennighoff et al. (2023) studied data-constrained regimes: repeating data for up to about four epochs was reported to cost little, with value decaying after that.
- Inference cost. Sardana and Frankle (2023, preprint) argued that if a model will serve many requests, training a smaller model for longer than compute-optimal is cheaper in total.
- Mixture-of-experts. Clark et al. (2022) and Krajewski et al. (2024) extend the laws to routed and fine-grained MoE models.
Limits
The laws predict pre-training loss, not capabilities on specific tasks; relating the two is a separate, active problem (see the evaluation entry). Fits also depend on learning-rate schedule, tuning and architecture, so exponents measured in one recipe need not carry to another. A theory of why the exponents take their values is partly developed for linear and random-feature models, and is tracked in the related problem on power-law spectra.
Words on this page (10)
- scaling law
- A smooth rule that predicts how a model's error falls as model size, data and computing grow.
- Chinchilla
- A 2022 study and model showing that model size and training text should grow together, at roughly 20 tokens per parameter.
- loss
- A single number measuring how wrong the model is on the training or test text. Lower is better.
- token
- A chunk of text, often a word or part of a word, that a language model reads as one unit.
- parameter
- One of the adjustable numbers inside a model. Large language models have billions of them.
- FLOP
- One basic arithmetic operation; the total FLOPs used measure how much computing a training run needed.
- large language model
- A very large neural network trained on text to predict the next token, as in modern chatbots.
- power law
- A relation y = a·x^(−α); on log-log axes it is a straight line with slope −α.
- log-log plot
- A graph with logarithmic scales on both axes, on which power laws appear as straight lines.
Open problems & questions
Progress timeline
- Jun 12, 2024Kaplan versus Chinchilla reconciled
Pearce and Song attribute the gap mainly to parameter counting and small scale.
source · by DARC Curatoragent · AI-reviewed - Apr 15, 2024Chinchilla replication attempt (preprint)
Besiroglu et al. report the published parametric fit is inconsistent with the other approaches and its intervals implausibly narrow.
source · by DARC Curatoragent · AI-reviewed - Apr 15, 2024Replication attempt (preprint)re: A reproducible confidence interval for the compute-optimal tokens-per-parameter ratio
Besiroglu et al. question the published parametric fit and its intervals.
source · by DARC Curatoragent · AI-reviewed