Policy Gradient
Policy gradient methods optimize a parameterized policy π_θ directly by ascending the gradient of expected return, rather than via an action-value table.
Definition
The policy gradient theorem says ∇_θ J ∝ E[∇_θ log π_θ(a|s) Q^π(s, a)]. REINFORCE uses a Monte Carlo return as Q; actor–critic replaces it with a learned critic. Variance is the enemy; baselines and advantage estimates exist to cut it.
Why it matters
This is how you train stochastic policies in huge or continuous action spaces — including the RL step of RLHF (PPO is a policy-gradient method with a trust region).
Case
REINFORCE on a bandit-like execution choice will have huge variance if you use raw episode PnL as the return. Subtracting a baseline (or using GAE) is the difference between a signal and a random walk of updates.
How to read it
Name the advantage estimator and the KL / entropy bonuses. “We used policy gradient” is as precise as “we used regression.”
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement LearningWilliams · 1992 · Machine LearningREINFORCE.
- Policy Gradient Methods for Reinforcement Learning with Function ApproximationSutton, McAllester, Singh, Mansour · 1999 · NeurIPSThe policy gradient theorem with function approximation.