Predicting benchmark accuracy of 7B+ models within 2 points beyond four multiple-choice tasks
Difficulty 48: the method's own extension to OLMo 2 32B missed by 5.8 points on average, and generative exact-match metrics are not smooth in loss, so a 2-point bar on 20 tasks needs new ideas. Application value 51: forecasting downstream value before a large run steers data and size choices. Leverage 37: underlies capability forecasting more broadly.
The problem
Can small models predict a big model's test scores within 2 points on 20 different tests, including coding and maths?
- 1Fit loss from model and data size
- 2Fit task score from loss
- 3Forecast the big model
- 4Extend to generative tasks
What is asked. Before spending millions on a large training run, a team would like to know how the finished model will score on tests such as coding, maths and general-knowledge benchmarks. A model ladder trains a series of small models costing together about 1% of the target, then predicts the target's scores in two steps: first fit a scaling law from model and data size to loss (the average prediction error), then fit a second curve from loss to task accuracy. The problem asks whether this predicts a 7B-or-larger model's score within 2 percentage points on at least 20 benchmarks, including generative tasks such as writing code that passes tests or maths answers checked by exact match, and on a model family other than the one used to build the method, with predictions published before the target is measured.
Built from
earlier problems → this oneThe problems this one is built from: progress or tools it needs (builds on), special cases that come first (generalizes), and weaker results on the way (implies). On the right, what builds on it.
- Start with
- Then
- Predicting benchmark accuracy of 7B+ models within 2 points beyond four multiple-choice tasksThis problem
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- The transformer architectureAlgorithm
- Adam, AdamW and the optimizer zooAlgorithm
- Step 2
- Step 3
- Language-model evaluation methodologyTheory · host
- Then
- Predicting benchmark accuracy of 7B+ models within 2 points beyond four multiple-choice tasksThe problem
State of the art
Bhagia et al. report that errors within 2 points were obtained on four ranked-classification tasks and that tasks with higher prediction error have higher variance across checkpoints. Standard loss power laws alone do not predict task accuracy. Observational scaling laws (Ruan et al., arXiv:2405.10938) take a different route, fitting across many existing public models. Schaeffer et al. (arXiv:2304.15004) argue that metric choice drives apparent emergence, which suggests generative exact-match tasks are the harder case. The paper's own appendix applies the ladder to OLMo 2 32B (trained on 6T tokens, ladder at about 0.45% of its compute) on seven multiple-choice tasks and reports a mean absolute error of 5.81 points, with ARC-Challenge off by 13.2, so the 2-point result does not yet carry to larger targets. DataDecide (Magnusson et al., arXiv:2504.11393) releases 25-corpus model suites up to 1B parameters and finds small-scale rankings predict the better data about 80% of the time, which is a ranking rather than an accuracy prediction. No ladder prediction validated on 20 or more tasks including generative ones was found.
Realistic applications
Forecasting benchmark results before launching a large run lets teams choose data mixes and sizes by expected downstream value instead of loss only.
What counts as solved
A public study with ladder code, small-model checkpoints and held-out target-model scores, showing absolute error of at most 2 points on at least 20 tasks including generative tasks, with the target predictions published before the target model is evaluated.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmarks: at least 20 tasks including generative ones, scored with the lm-evaluation-harness at pinned versions. Metric: absolute accuracy error between predicted and measured score of the target model, per task. Threshold: at most 2 points on at least 18 of 20 tasks, and predictions committed (hashed and timestamped) before the target is evaluated. Budget: ladder runs of about 1% of the target's training compute (for a 7B model on 4T tokens, about 1.7x10^23 FLOPs in total, so about 1.7x10^21 FLOPs for the ladder); at least 3 ladder seeds.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Method development, then ladder training and held-out target evaluation.
Bottlenecks
ranked by votes- #1TheoryNon-smooth generative metrics
Exact-match and pass@k metrics are not smooth in loss, so the two-step fit may fail.
Community grades
Sign in to add your own grade.
Connected problems
Progress
- Dec 5, 2024Model ladders publishedsource
Predicted 7B and 13B accuracy within 2 points on four tasks.