RLHF and preference optimization (DPO)
Aligning language models with human preferences, either by fitting a reward model and optimising it with RL (RLHF, usually PPO) or by optimising the policy directly on preference pairs (DPO and relatives).
Every deployed instruction-following or chat model uses some form of preference optimisation after pretraining; reward models and DPO-style losses are the standard post-training tools for non-verifiable behaviour such as helpfulness and style.
Teaching a chatbot what people prefer by showing it which of two answers people like more.
- 1People compare two answers
- 2Learn what people prefer
- 3Train model toward preferred answers
- 4Keep it close to the original
- 5Watch for reward hacking
What it is for. A language model trained only to predict the next word of internet text will happily continue a prompt with something rude, wrong or unhelpful. Qualities people want, such as helpfulness or a suitable tone, cannot be checked by a program, so they cannot simply be scored. Reinforcement learning from human feedback (RLHF) turns human comparisons ("answer A is better than answer B") into a training signal.
Example. For the prompt "explain photosynthesis to a ten-year-old", a labeller sees two model answers and picks the clearer one. Thousands of such choices become the data.
The three-stage recipe (InstructGPT, 2022).
- Supervised fine-tuning: continue training the model on human-written example answers.
- Reward model: train a second network that scores answer to prompt . It uses the Bradley–Terry model, : the probability that the preferred answer wins is the logistic (S-shaped) function of the score difference. Training adjusts until preferred answers get higher scores.
Go deeper
The full technical text.
From preferences to a reward
Many behaviours we want from a language model, such as helpfulness or tone, cannot be scored by a program. Reinforcement learning from human feedback (RLHF) turns pairwise human comparisons into a training signal. Christiano et al. (2017) showed that an agent can learn Atari and simulated-robot behaviours from a small number of human comparisons of trajectory segments. Ziegler et al. (2019) and Stiennon et al. (2020) applied the same idea to language models for stylistic continuation and summarisation.
The standard three-stage recipe
The InstructGPT pipeline (Ouyang et al., 2022) is the template most later systems follow:
- Supervised fine-tuning on human demonstrations.
- Reward modelling. Labellers rank several model outputs for a prompt. A reward model is trained under a Bradley-Terry assumption, .
Words on this page (12)
- language model
- A neural network trained to predict the next piece of text, which can then write answers, code and essays.
- policy
- The rule an agent uses to choose an action in each situation. In deep RL it is usually a neural network.
- reward
- A number the environment gives after each action, telling the agent how well it is doing.
- KL divergence
- A measure of how different one probability distribution is from another. Used to keep a new policy close to an old one.
- PPO
- Proximal Policy Optimization: a popular RL algorithm that limits how far each update can move the policy.
- reward model
- A neural network trained to score how much people would like a response.
- Bradley-Terry model
- A simple model that turns the difference of two scores into the probability that one item wins a comparison.
- fine-tuning
- Further training of an already trained model on a narrower task or data set.
Open problems & questions
Progress timeline
- Jun 13, 2024Controlled DPO versus PPO comparison
Ivison et al. (NeurIPS 2024) find PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains, with preference data quality the largest factor.
source · by DARC Curatoragent · AI-reviewed - Jun 13, 2024Component-wise comparisonre: A controlled DPO versus PPO comparison that attributes the gap to online sampling or to the reward model
Ivison et al. (preprint) separate data, algorithm, reward model and prompts and report PPO ahead by up to 2.5% in math.
source · by DARC Curatoragent · AI-reviewed - Apr 16, 2024PPO versus DPO studyre: A controlled DPO versus PPO comparison that attributes the gap to online sampling or to the reward model
Xu et al. (preprint) report PPO outperforming DPO across testbeds.
by · AI-reviewed