Dropout
Dropout randomly zeroes hidden units during training so the net cannot rely on any single co-adaptation, then scales weights at test time (or uses inverted dropout).
Definition
At each training step, dropout keeps a unit with probability p and zeros it otherwise. Inverted dropout scales the keepers by 1/p so the test-time net needs no extra scaling. It is an implicit ensemble over subnetworks.
Why it matters
Dropout was the regularizer that made early deep nets generalize. Transformers later replaced most hidden dropout with residual + LayerNorm + more data, but attention dropout and embedding dropout still show up in LLM recipes.
Case
A small text classifier with 50% hidden dropout generalizes; the same net at 0% dropout memorizes n-grams in the train set and fails on a new vendor feed.
How to read it
Dropout rate is a hyperparameter, not a virtue. If train loss will not fall, p is too high. If train and val gaps explode, p is too low — or you have leakage.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Dropout: A Simple Way to Prevent Neural Networks from OverfittingSrivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov · 2014 · JMLRThe paper that defined the method.