Vanishing Gradient
Vanishing gradients are when backprop multiplies many |Jacobian| < 1 factors so early layers receive ~0 update — the reason plain deep RNNs and tanh stacks died.
Definition
The chain rule through depth T or time T is a product of T matrices. If their norms are < 1, the product collapses; if > 1, it explodes. LSTM gates, residuals, ReLU, and LayerNorm are architectural answers. Gradient clipping is a bandage for explosions.
Why it matters
If early layers do not move, you do not have a deep model — you have a shallow model with dead decoration.
Case
A 20-layer tanh MLP on tabular data: train loss falls via the last layers only. Residual + ReLU revives the bottom.
How to read it
Log gradient norms by layer. Silent bottoms are vanishing; NaNs are exploding. Fix the backward graph before buying more data.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Learning Long-Term Dependencies with Gradient Descent is DifficultBengio, Simard, Frasconi · 1994 · IEEE Transactions on Neural NetworksWhy vanilla RNNs vanish.