Gradient Descent
Gradient descent updates parameters against the gradient of a loss: θ ← θ − η ∇_θ L. Stochastic and mini-batch variants make the method tractable on large datasets.
Definition
Gradient descent is first-order minimization: follow the negative gradient of a differentiable loss. Full-batch GD uses the whole dataset; stochastic gradient descent (SGD) uses one example or a mini-batch, injecting noise that often helps generalization. The learning rate η and schedule dominate whether the run converges or diverges.
Why it matters
Model quality is not just architecture. A Transformer with a bad schedule underperforms a smaller net with a sane one. In research and in desk ML, most “the model doesn’t work” tickets are optimizer / LR / batch-size tickets.
Case
Double the learning rate on a language-model pretrain and the loss spikes to NaN. Halve it and you waste a week on a plateau. Warmup plus cosine decay is the usual fix — not a new architecture.
How to read it
Plot train loss, grad norm, and a held-out metric on the same clock. If train loss falls and the metric does not, you are optimizing the wrong objective or leaking the test set.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Some studies in machine learning using the game of checkersSamuel · 1959 · IBM JournalEarly operational use of iterative parameter updates from experience.
- Adam: A Method for Stochastic OptimizationKingma, Ba · 2015 · ICLRThe default adaptive method most teams actually run.