Methods that improve a policy directly by following a sampled estimate of the gradient of expected return. Proximal Policy Optimization (PPO) is the default on-policy variant in games, robotics and language models.
PPO is the optimiser behind OpenAI Five, the original RLHF pipelines for instruction-following models, and most RL baselines; GRPO and other reasoning-model recipes are modifications of its clipped objective.
Teach a decision-maker by nudging it toward actions that worked, in small safe steps.
1Try actions, collect rewards
2Estimate which actions beat average
3Nudge policy toward better actions
4Clip the change to stay safe
5Repeat with fresh experience
What it is for. Reinforcement learning (RL) trains an agent, a program that picks actions, by trial and error: it acts, receives a number called a reward, and adjusts itself to collect more reward over time. A robot arm learning to grasp or a game agent learning to win are typical cases. The policy is the agent's rule for choosing actions. Here it is a neural network, a large function with adjustable parameters θ fitted by gradient descent, that outputs a probability for each action.
The goal. The agent maximises the expected discounted return J(θ)=E[∑. An expected value is a probability-weighted average over everything that could happen. The discount factor (slightly below 1) makes rewards far in the future count a little less, like interest in reverse.
Go deeper
The full technical text.
The problem they solve
A reinforcement learning agent picks actions a in states s with a policy πθ(a∣ and receives rewards. The goal is to maximise the expected discounted return . Value-based methods such as Q-learning learn how good actions are and act greedily on that. Policy gradient methods instead adjust the parameters of the policy itself, which handles continuous actions and stochastic policies naturally.
Exponential-time lower bound for softmax policy gradient
Li, Wei, Chi, Gu and Chen construct a three-action MDP on which exact-gradient softmax policy gradient needs a number of iterations doubly exponential in the effective horizon (COLT 2021; Mathematical Programming).
Li et al. exhibit a three-action MDP requiring exponentially many iterations (preprint).
Discussion
t
γt
rt
]
γ
The policy gradient. Gradient ascent climbs J by moving θ along the gradient, the vector of partial derivatives (the slope in every parameter direction at once). The difficulty is that J depends on the environment, which cannot be differentiated. The policy gradient theorem (Sutton et al., 1999) avoids this:
∇θJ=E[∇θlogπθ(a∣s)A(s,a)].
In words: for each action actually taken, nudge the parameters to make that action more likely if it turned out better than expected (positive advantage A) and less likely if worse. The advantage is the action's return minus a baseline, an estimate V(s) of how good the situation was anyway. Subtracting the baseline leaves the average direction unchanged but makes the noisy estimate much steadier. REINFORCE (Williams, 1992) is the simplest version, using sampled returns.
Why PPO. A step that is too large can change the policy so much that performance collapses, and the data collected under the old policy no longer describes the new one. TRPO (2015) limits each step by the KL divergence, a measure of how different two probability distributions are. PPO (Schulman et al., 2017) gets a similar effect cheaply: it computes the ratio ρ between the new and old probability of each action and clips it to [1−ϵ,1+ϵ] (the paper used ϵ=0.2), so the update gains nothing by moving further. Several passes of updates can then reuse one batch of experience.
In practice. Engstrom et al. (2020) found that much of PPO's advantage over TRPO on robot-control benchmarks came from code-level details (reward scaling, initialisation, learning-rate schedules) rather than the clipping rule. PPO trained OpenAI Five and the RL stage of InstructGPT; CleanRL and Stable-Baselines3 hold reference implementations.
s
)
J(θ)=Eπθ[∑tγtrt]
θ
The policy gradient theorem
Sutton, McAllester, Singh and Mansour (NeurIPS 1999) showed that the gradient of J can be written without differentiating the unknown state distribution:
∇θJ(θ)=Eπθ[∇θlogπθ(a∣s)Qπθ.
Williams's REINFORCE (1992) is the Monte Carlo estimator of this expression, using sampled returns in place of Q. Its variance is high. Subtracting a state-dependent baseline, usually a learned value function V(s), leaves the gradient unbiased and reduces variance; the weight on ∇logπ becomes an advantageA(s,a)=Q(s,a)−V(s). Generalised Advantage Estimation (Schulman et al., 2015) trades bias against variance with a parameter λ and is the standard advantage estimator in actor-critic code.
Trust regions and PPO
A plain gradient step can change the policy so much that performance collapses and the data collected under the old policy no longer describes the new one. Trust Region Policy Optimization (TRPO, Schulman et al., 2015) maximises a surrogate objective subject to a bound on the KL divergence between old and new policy, with a monotonic-improvement argument behind it. PPO (Schulman et al., 2017) replaces the constrained problem with a cheap clipped surrogate. With the probability ratio ρt(θ)=πθ(at∣st)/πθold(at∣st) it maximises
LCLIP(θ)=Et[min(ρtAt,clip(ρt,1−ϵ,1+ϵ),
which removes the incentive to move the ratio outside [1−ϵ,1+ϵ], so several epochs of minibatch updates can reuse one batch of rollouts. The paper's experiments used ϵ=0.2 as the best-performing value.
What the algorithm really is
Engstrom et al. (2020) found that much of PPO's advantage over TRPO on continuous-control benchmarks comes from code-level choices such as value-function clipping, reward scaling, orthogonal initialisation and learning-rate annealing, not from the clipping rule itself. Huang et al. documented 37 implementation details that affect results. The consequence is that "PPO" in a paper and "PPO" in a library can differ materially, and reproduction claims need the code, not only the objective.
Where it is used
PPO trained OpenAI Five (Dota 2, 2019) and was the reinforcement-learning stage of InstructGPT. Open reference implementations include CleanRL and Stable-Baselines3. Theory lags practice: tabular and softmax policy gradient have global convergence guarantees under assumptions on distribution mismatch (Agarwal et al., 2019), and softmax policy gradient can need exponentially many iterations in the effective horizon (Li et al., 2021), but guarantees for the clipped objective used in practice with neural networks are limited to overparametrised variants (Liu et al., 2019).
A large function with many adjustable numbers (weights), fitted to data by gradient descent; it maps inputs such as a game screen to outputs such as action probabilities.
A large function with many adjustable numbers (weights), fitted to data by gradient descent; it maps inputs such as a game screen to outputs such as action probabilities.
What to read before Policy gradient methods and PPO (left), and what builds on it: entries that depend on it and the open problems it hosts (right). Every box links to its page.