Batch Normalization
Batch normalization re-centers and re-scales layer inputs using mini-batch statistics, then learns a scale and shift, reducing internal covariate shift and allowing higher learning rates.
Definition
BN computes μ, σ over the current mini-batch for each channel, normalizes, then applies learned γ, β. At inference it uses running averages. It couples examples inside a batch, which is why batch size becomes a silent hyperparameter.
Why it matters
BN made deep ConvNets trainable at higher LR and less sensitive to init. It is the wrong tool when the batch is tiny, when you need deterministic per-example behavior, or in many sequence models — that is why LayerNorm won in Transformers.
Case
A detection model trained with batch 32 and evaluated with batch 1 will shift if you forget to switch to running stats. mAP drops and it looks like a “deployment bug.” It is BN mode.
How to read it
Know train vs eval mode. If metrics jump when batch size changes, you are looking at BN, not at a new market regime.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftIoffe, Szegedy · 2015 · ICMLOriginal BN paper.