Q-Learning
Q-learning is an off-policy TD method that learns action values Q(s, a) toward the greedy target r + γ max_a′ Q(s′, a′), without needing the behavior policy to be optimal.
Definition
Watkins’ Q-learning updates Q(s, a) ← Q(s, a) + α [r + γ max_a′ Q(s′, a′) − Q(s, a)]. It is off-policy: you can explore randomly while learning the greedy values. Function approximation (DQN) makes it scale and also makes it unstable without replay and a target network.
Why it matters
This is the value-based backbone. If you understand the max operator and off-policy data, you understand why DQN needs replay and why overestimation happens (Double Q).
Case
Tabular Q-learning solves a small gridworld. On Atari, a table is impossible; DQN uses a CNN as Q and a replay buffer so successive updates are less correlated.
How to read it
Off-policy is not “free data.” If the behavior policy never visits a region, Q is fiction there. In trading logs that is most of the action space.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Q-learningWatkins, Dayan · 1992 · Machine LearningThe Q-learning paper.