Do reward-model overoptimization laws hold for modern RLHF and DPO-style training?
Difficulty 30: a reproducible measurement programme with a known protocol; the cost is compute and a trustworthy gold reward. Application value 40: guidance on early stopping and reward-model size. Leverage 30: sharpens post-training practice without unlocking new capabilities; below the RLVR question (52).
The problem
How far can you push an AI toward a score before the score stops meaning what you wanted?
- 1Train against a proxy score
- 2Measure the true score
- 3Score rises, then falls
- 4Fit curves to predict the peak
- 5Test on today's models
Why it matters. In RLHF a language model is optimised against a reward model, a network trained to imitate human preferences. The reward model is only an approximation, and pushing hard on an approximation eventually exploits its errors: the reward score keeps rising while real quality falls. This is Goodhart's law. Knowing when the turn happens tells practitioners how long to train and how large a reward model to use.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- Policy gradient methods and PPOAlgorithm
- The transformer architectureAlgorithm
- Step 3
- RLHF and preference optimization (DPO)Algorithm · host
- Then
- Do reward-model overoptimization laws hold for modern RLHF and DPO-style training?The problem
State of the art
The 2022 study used synthetic gold labels and smaller models than current assistants. It is widely cited as motivation for KL penalties and early stopping, but repeated measurement at modern scale and for direct-alignment losses is thin in the literature. RewardBench 2 (2025, preprint) reports that reward-model benchmark accuracy correlates with downstream best-of-N and PPO performance, which is related but does not give the overoptimization curve.
Realistic applications
Gives practical guidance on how long to optimise against a learned reward and how large a reward model is needed, which affects the cost and safety of post-training.
What counts as solved
Published fits with held-out extrapolation on at least two policy scales (at least 7B parameters) and both an RL method and a direct-alignment loss, with the functional forms either confirmed with parameter uncertainty or replaced by a better-fitting form.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Setup: a fixed gold reward model, proxy reward models of at least three sizes trained on gold-labelled preference data, and policies of at least 7B parameters optimised by (a) best-of- up to of 1,000 or more, (b) PPO or GRPO with a KL penalty, (c) DPO with varied and dataset size. Metric: gold reward versus with fits of the forms reported by Gao et al. and alternatives, scored by held-out extrapolation error (fit on , test on ). Seeds: at least 3 per setting for confidence bands. Compute: report GPU-hours; the original study scale is not assumed.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Experiments with large models and a faithful gold reward dominate; some theory in choosing the functional form.
Bottlenecks
ranked by votes- #1EmpiricalFaithful gold reward
A synthetic gold model only represents human preferences to the extent it is trustworthy; results may not transfer to real human labels.
Community grades
Sign in to add your own grade.
Connected problems
- This problem is related to A controlled DPO versus PPO comparison that attributes the gap to online sampling or to the reward model
Progress
- Oct 19, 2022Overoptimization scaling lawssource
Synthetic gold-versus-proxy setup shows smooth coefficient scaling for best-of-n and RL.