A controlled DPO versus PPO comparison that attributes the gap to online sampling or to the reward model
Difficulty 25: a controlled experiment with known ingredients; the barriers are matched compute and confounder control. Application value 42: concrete for teams choosing between offline losses and online RL. Leverage 28: a methodological result within post-training.
The problem
Two ways to teach chatbots what people like: which is better, and why?
- 1Fix model, data and prompts
- 2Train with DPO and with PPO
- 3Add online variants as controls
- 4Match compute, repeat seeds
- 5Compare against noise
Why it matters. Teams that align language models with human preferences choose between two families. DPO trains on a fixed dataset of preferred and rejected answers with a simple supervised loss. PPO-based RLHF trains a reward model, then repeatedly samples fresh answers from the current model, scores them and updates the model. PPO costs much more engineering and compute. If it is genuinely better, the cost is worth paying; if not, the effort is wasted.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- Policy gradient methods and PPOAlgorithm
- The transformer architectureAlgorithm
- Step 3
- RLHF and preference optimization (DPO)Algorithm · host
- Then
- A controlled DPO versus PPO comparison that attributes the gap to online sampling or to the reward modelThe problem
State of the art
Two preprints favour PPO: Xu et al. report PPO surpassing other alignment methods across their testbeds including code competitions; Ivison et al. report PPO ahead of DPO by up to 2.5% in math and 1.2% in general domains, with preference-data quality the largest factor, followed by algorithm choice, reward-model quality and extra prompts. Preference data, reward models, evaluations and prompt sets vary between studies and mix several effects, and gains are small relative to evaluation noise on some benchmarks. Which factor explains the gap is not isolated.
Realistic applications
Reduces wasted compute in post-training by showing whether the cost of online RL is justified over simpler offline losses.
What counts as solved
A matched-budget study with at least 3 seeds per arm in which the arm differences are resolved against seed variance and replicated by an independent group on a second base model.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmark: the evaluation suite of Ivison et al. (open-instruct evaluations) or a stated equivalent with at least 8 tasks covering instruction following, reasoning, math, code and safety, plus a human or judge-model win rate. Data: one public preference dataset (for example UltraFeedback) used identically by all arms. Compute: equal total GPU-hours including generation, stated for each arm. Seeds: at least 3 training seeds per arm; the claim holds only if the difference in aggregate score exceeds twice the standard deviation across seeds. Repos: open-instruct and TRL implementations.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Mainly compute and careful engineering to match arms; evaluation noise is the empirical limit.
Bottlenecks
ranked by votes- #1EngineeringMatching implementations
Online RL and offline losses differ in infrastructure; matched hyperparameter tuning and generation costs are hard to equalise.
Community grades
Sign in to add your own grade.
Connected problems
- This problem is related to Do reward-model overoptimization laws hold for modern RLHF and DPO-style training?