Proximal Policy Optimization
PPO is a policy-gradient algorithm that clips the probability ratio so each update stays close to the previous policy, giving much of TRPO’s stability with first-order SGD.
Definition
Schulman et al. (2017) maximize a clipped surrogate: the probability ratio r_t(θ) is clipped into [1−ε, 1+ε] so large policy steps are penalized. Combined with GAE, it became the default on-policy algorithm for games and for early RLHF.
Why it matters
If a paper says they aligned a chatbot with RL, the algorithm was probably PPO (or a later DPO that skips the RL loop). Knowing the clip and the KL-to-reference is how you debug a reward-hacked run.
Case
OpenAI used PPO on robotics and games, then on InstructGPT. Unclipped updates with a misspecified reward make the policy collapse to high-reward garbage text.
How to read it
Report clip ε, KL penalty, and reward scale. If reward can be hacked by verbosity or sycophancy, PPO will find it — that is the algorithm working.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Proximal Policy Optimization AlgorithmsSchulman et al. · 2017 · arXivPPO.