Adam Optimizer
Adam is an adaptive first-order optimizer that keeps exponential moving averages of the gradient and its square, giving per-parameter step sizes.
Definition
Adam updates m̂_t / (√v̂_t + ε) with bias correction on the first and second moments. AdamW decouples weight decay from the adaptive step, which is the form used in almost every LLM run. Default β = (0.9, 0.999) is a starting point, not a law.
Why it matters
Most production trains are AdamW plus a warmup/cosine schedule. Knowing what Adam actually adapts — and that it can still diverge or find sharp minima — keeps you from treating the optimizer as magic.
Case
A fine-tune with AdamW 2e-4 and no decay melts a pretrained LM. Drop to 2e-5, add warmup, and the same data becomes a useful adapter. Optimizer + LR, not “the base model is bad.”
How to read it
Log LR, weight-decay mode (Adam vs AdamW), and whether you are training full weights or adapters. Compare wall-clock to a well-tuned SGD baseline on small problems before declaring Adam mandatory.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Adam: A Method for Stochastic OptimizationKingma, Ba · 2015 · ICLROriginal Adam.
- Decoupled Weight Decay RegularizationLoshchilov, Hutter · 2019 · ICLRAdamW — the variant people actually use.