Kullback–Leibler Divergence
KL divergence KL(P‖Q) = E_P[log dP/dQ] is the expected extra log-loss from using Q when the truth is P — the loss behind cross-entropy training and many variational methods.
Definition
KL is ≥ 0 and zero iff P = Q (a.e.). It is not a symmetric distance and does not obey the triangle inequality. Cross-entropy = entropy(P) + KL(P‖Q). VAEs and many RL penalties are KL terms.
Why it matters
When a paper says they ‘match distributions,’ ask which KL direction. KL(P‖Q) and KL(Q‖P) behave differently in the tails — relevant for risk and for generative models.
Case
A risk-neutral measure Q that is far from P in KL is a large risk premium. A language model minimizing cross-entropy is minimizing KL to the data distribution (plus entropy).
How to read it
State the direction and the support. If Q puts zero mass where P has mass, KL is infinite — your variational family is wrong.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- On Information and SufficiencyKullback, Leibler · 1951 · Annals of Mathematical StatisticsThe original KL paper.