Layer Normalization
Layer normalization standardizes activations across features for each example, not across the batch — the stabilizer that made Transformers trainable.
Definition
LN computes mean and variance over the hidden dimension of a single token (or example), then applies γ, β. It does not depend on batch size, so it works with batch 1 and with variable-length sequences.
Why it matters
Pre-norm vs post-norm LayerNorm is one of the highest-leverage Transformer training choices. RMSNorm is a cheaper cousin used in LLaMA-style models. If a train diverges at depth, look here before inventing a new attention variant.
Case
A 24-layer post-norm Transformer diverges at LR 3e-4; the same stack with pre-norm and LN trains. The data did not change — the backward scale did.
How to read it
State where LN sits (before or after residual add) and whether it is LN or RMSNorm. Those two bits explain more failed runs than most architecture tweets.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Layer NormalizationBa, Kiros, Hinton · 2016 · arXivThe normalization used throughout modern sequence models.