Reinforcement Learning from Human Feedback
RLHF fits a reward model to human (or AI) preference comparisons, then optimizes a language model against that reward, usually with a KL penalty back to a reference policy.
Definition
Christiano et al. (2017) learned rewards from pairwise preferences for RL agents. Ouyang et al. (InstructGPT, 2022) applied the stack to LMs: SFT on demos, reward model on rankings, PPO (or later DPO) to optimize. The KL term stops the policy from drifting into reward-hacking gibberish.
Why it matters
This is why chat models follow instructions and refuse some harmful asks. It is also why they sycophantically agree and why reward-model misspecification becomes a product incident.
Case
InstructGPT was preferred to a 100× larger GPT-3 on human evals of helpfulness. The win was the alignment stack, not raw pretrain scale.
How to read it
Ask who labeled, on what distribution, and what the KL budget was. “We did RLHF” without those three facts is a press release.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Deep Reinforcement Learning from Human PreferencesChristiano et al. · 2017 · NeurIPSPreferences → reward model → RL.
- Training language models to follow instructions with human feedbackOuyang et al. · 2022 · NeurIPSInstructGPT.