Cross-Entropy Loss
Cross-entropy measures how well a predicted distribution q matches a target distribution p. For one-hot labels it reduces to −log q(y), the usual classification and language-model loss.
Definition
H(p, q) = −Σ p log q. With a one-hot target this is negative log-likelihood of the correct class. Language models minimize token-level cross-entropy, reported as bits-per-byte or perplexity exp(H). It pairs with a softmax (or sigmoid for binary) output.
Why it matters
If the loss does not match the decision you care about, the trained model will optimize the wrong thing — common in imbalanced labels, ranking, and next-token prediction used as a proxy for “truth.”
Case
A news classifier with 99% negatives can get low cross-entropy by always predicting “no event.” Accuracy looks great; recall on the rare class is zero. Class weights or a ranking loss are the fix, not a bigger net.
How to read it
Report the loss *and* the operating metric (F1, calibration, PnL). Falling cross-entropy with a collapsing rare-class recall means the objective and the job have diverged.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- On the importance of initialization and momentum in deep learningSutskever, Martens, Dahl, Hinton · 2013 · ICMLPractical deep training under log-loss / NLL with modern init and momentum.