Do RL compute-performance curves for LLMs extrapolate across models, domains and recipes?
Difficulty 32: a method and a recipe exist; the open work is replication on other model families and domains. Comparable to compute-optimal-tokens-per-parameter-confidence-interval (30). Application value 50: predictable RL scaling would guide compute planning. Leverage 35: limited to post-training practice.
The problem
Can small practice runs predict how good a huge reinforcement-learning run will be?
- 1Run small RL experiments
- 2Fit an S-shaped curve
- 3Predict a much bigger run
- 4Run it and compare
- 5Check other models and tasks
Why it matters. Before a lab trains a large language model it can predict the final loss from small runs, because pretraining loss falls as a power law in compute (a straight line on a log-log plot). RL post-training has no comparable rule, so a group cannot tell from a small pilot whether a 100,000 GPU-hour run (a GPU-hour is one graphics processor working for one hour) will pay off.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- Policy gradient methods and PPOAlgorithm
- The transformer architectureAlgorithm
- Step 3
- Step 4
- RL with verifiable rewards and GRPO for reasoning modelsFrontier · host
- Then
- Do RL compute-performance curves for LLMs extrapolate across models, domains and recipes?The problem
State of the art
A 2025 preprint (more than 400,000 GPU-hours) fits sigmoidal compute-performance curves, reports that some design choices change the asymptote while others (loss aggregation, normalisation, curriculum, off-policy algorithm) mainly change compute efficiency, and demonstrates predicting validation performance of one run scaled to 100,000 GPU-hours. It is a single study from one group, with one recipe (ScaleRL) shown to extrapolate; independent replications, other model families, other domains and sparse (mixture-of-experts) models are not established.
Realistic applications
Lets labs and academic groups decide whether to launch a large RL run from small pilot runs, and compare algorithms by asymptote rather than by performance at one budget.
What counts as solved
Independent replication of the extrapolation test (below) on at least two base-model families not in the original study, on at least two task domains, with the prediction interval covering the held-out result.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Dataset and metric: held-out validation accuracy (pass@1, averaged over at least 16 samples per problem) on a stated maths set and a stated code set. Test: fit the curve on runs up to (a proposed value is 2,000 GPU-hours or less so that groups outside large labs can take part), pre-register the prediction and its interval for , then run to . Pass if the observed value lies in the interval for at least two base-model families and two domains. Seeds: at least 3 seeds at the fit scale to estimate run-to-run variance. Publish training code and the exact hyperparameters, since recipe details determine the asymptote.
Claims under verification
+ add a claimWhere the work is
Dominated by the GPU-hours needed to run enough RL curves; engineering matters for stable recipes.
Bottlenecks
ranked by votes- #1ComputeCost of large-scale RL runs
Validating extrapolation needs runs at several times the fit scale, which is tens of thousands of GPU-hours per point and out of reach for most academic groups.
Community grades
Sign in to add your own grade.
Connected problems
Progress
- Oct 15, 2025Sigmoidal RL scaling studysource
A 400,000 GPU-hour preprint fits compute curves and predicts a 100,000 GPU-hour run.