Backpropagation
Backpropagation computes gradients of a scalar loss with respect to every weight by applying the chain rule backwards through the computational graph.
Definition
Backpropagation is reverse-mode automatic differentiation on a neural net. A forward pass caches activations; a backward pass multiplies local Jacobians so each parameter receives ∂L/∂θ. Without it, training deep models by finite differences is computationally hopeless.
Why it matters
Every optimizer you actually run — SGD, Adam, Lion — is only as good as the gradient. Vanishing or exploding backprop is why plain RNNs failed, why residuals and LayerNorm exist, and why mixed precision needs loss scaling.
Case
A 20-layer tanh MLP trained on a classification task will often show near-zero gradients in early layers. The architecture can represent the function; backpropagation cannot deliver a usable signal. Residual connections change the backward graph, not just the forward look.
How to read it
Watch gradient norms by layer, not only the loss. If early layers are silent, the model is not learning features — it is polishing the head. Fix the backward path before adding more data.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Learning representations by back-propagating errorsRumelhart, Hinton, Williams · 1986 · NatureCanonical statement of backprop for multilayer networks.