Does RLVR create new reasoning ability or only sharpen the base model? A pass@k protocol that separates the two
Difficulty 50: experiments are feasible but published results conflict and depend on base model, recipe, training length and sampling budget, so a decisive protocol is research-grade; level with optimizer-speedup-over-tuned-adamw-1b-scale (50). Application value 60: labs split compute between pretraining, RL and distillation partly on this answer. Leverage 52: it frames how RL on reasoning models is interpreted, a subfield-shaping question.
The problem
When AI practises on checkable problems, does it learn new tricks or just get more reliable at old ones?
- 1Pick base model and RL recipe
- 2Sample thousands of answers each
- 3Check answers and reasoning
- 4Count problems only RL solves
- 5Repeat across seeds and models
Why it matters. Labs spend large amounts of computation on reinforcement learning with verifiable rewards (RLVR) after pretraining. If RL teaches a model to solve problems it previously could not, more RL compute buys new ability. If RL only makes the model pick, more reliably, solutions it could already produce occasionally, the ability has to come from the base model or from distillation (training on a stronger model's outputs).
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- Policy gradient methods and PPOAlgorithm
- The transformer architectureAlgorithm
- Step 3
- Step 4
- RL with verifiable rewards and GRPO for reasoning modelsFrontier · host
- Then
- Does RLVR create new reasoning ability or only sharpen the base model? A pass@k protocol that separates the twoThe problem
State of the art
Evidence conflicts. Yue et al. (arXiv preprint, 2025) evaluate pass@ with up to 128 to 1024 on math, code and visual benchmarks and report that RL-trained models beat base models at small but are overtaken at large ; on AIME 2024 at , 13.3% of problems were solved only by the base model and 0.0% only by the RL model, though on MATH500 at GRPO's pass@256 (97.2) was slightly above the base model's (96.2), so the deficit is not universal. They find distilling from a stronger teacher does extend the boundary. ProRL (NVIDIA, preprint) reports that prolonged RL with KL control and reference-policy resets solves problems the base model fails regardless of attempts. "The Invisible Leash" (preprint) finds on question-level matching that lost verified successes outweigh gained ones at the tested budgets. Open issues: pass@ on short-answer maths can count lucky guesses; larger is expensive; base-model pretraining already influences results (Liu et al., 2025, preprint).
Realistic applications
Decides whether scaling RL compute can produce capabilities beyond pretraining or whether capability must come from better base models and distillation, which shapes how labs allocate compute between pretraining, RL and distillation. No direct product use.
What counts as solved
An independently reproduced, pre-registered evaluation on at least two base-model families showing, with confidence intervals over problems and over at least three RL training seeds, either a significant positive set of base-unsolved problems solved by with verified reasoning, or a bound showing none at the stated budget.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmarks: AIME 2024 and 2025, a held-out olympiad set and a code set with hidden unit tests (for example LiveCodeBench), plus a contamination check against the base model's training data. Protocol: sample responses per problem from and from at temperature 0.6 and top-p 0.95 (the setting used by Yue et al.), report unbiased pass@ for , and count problems with solved by . Accept a correct-reasoning label only after automatic answer checks plus manual or model-assisted chain-of-thought review of a pre-registered sample. Seeds: at least 3 independent RL training runs per recipe with bootstrap confidence intervals over problems. Compute: report GPU-hours for RL and sampling; evaluation alone is on the order of to sampled responses per model on the maths set, so budget accordingly.
Claims under verification
+ add a claimWhere the work is
Mostly large-scale sampling and training runs plus careful experimental design; the theory content is about defining expansion versus sharpening.
Bottlenecks
ranked by votes- #1ComputeSampling cost of large pass@k
Resolving rare solves needs 1,000 or more samples per problem across problem sets, base and RL models and training seeds, which is millions of long reasoning traces per configuration.
Community grades
Sign in to add your own grade.
Connected problems
- This problem is related to Do RL compute-performance curves for LLMs extrapolate across models, domains and recipes?
Progress
- Jul 20, 2025Support-dynamics analysissource
Question-level matching finds shrinkage of verified support outweighs expansion at tested budgets (preprint).
- May 30, 2025ProRL evidence for expansionsource
Prolonged RL is reported to solve problems that base models fail regardless of sampling (preprint).
- Apr 18, 2025Pass@k evidence against expansionsource
Yue et al. report base models matching or exceeding RL models at large k (preprint).