Scaling Laws
Scaling laws are empirical power laws relating language-model loss to parameter count, data, and compute, used to plan pretraining rather than guess.
Definition
Kaplan et al. (2020) fit L(N, D, C) as power laws. Hoffmann et al. (Chinchilla, 2022) argued that under a compute budget, many “big” models were undertrained — you should scale tokens with parameters, not only parameters.
Why it matters
This is how labs decide runs that cost millions of dollars. If your 7B model was trained on too few tokens, “we need 70B” may be the wrong conclusion.
Case
A 10B model trained Chinchilla-optimally can beat a 100B model trained on too little data at the same FLOP budget. Size headlines mislead; tokens/parameter do not.
How to read it
State the compute budget, tokens, and parameters. IsoFLOP curves beat single-run folklore. Laws are fits, not theorems — they break when data quality or the objective changes.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Scaling Laws for Neural Language ModelsKaplan et al. · 2020 · arXivKaplan scaling laws.
- Training Compute-Optimal Large Language ModelsHoffmann et al. · 2022 · NeurIPSChinchilla.