Regularization
Regularization is any constraint that trades train fit for expected live error: weight decay, dropout, early stopping, data augmentation, or a simpler hypothesis class.
Definition
Regularization shrinks the effective hypothesis class. L2 weight decay penalizes ‖θ‖²; dropout randomly zeros units; early stopping treats optimization time as a capacity knob; label smoothing and augmentation change the data distribution. The point is not a prettier train curve.
Why it matters
Unregularized high-capacity models interpolate the sample. In language and vision that can still generalize (overparameterized regime); in small financial panels it usually does not.
Case
The same MLP with weight decay 0.01 and early stopping on a 2012–2018 validation window keeps a residual momentum signal. Without them it memorizes ticker IDs via leaked features.
How to read it
State the regularizer and the selection rule. If the only regularizer is “we picked the seed that looked best on test,” you regularized toward the test set.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Dropout: A Simple Way to Prevent Neural Networks from OverfittingSrivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov · 2014 · JMLRDropout as explicit capacity control.