Xavier Initialization
Xavier/Glorot initialization scales initial weights so variance is preserved through a layer — the default that made deep tanh/sigmoid nets trainable before BatchNorm.
Definition
Glorot & Bengio (2010): draw weights with variance 2/(n_in + n_out) so forward activations and backward gradients keep comparable scale. He init (2/n_in) is the ReLU cousin. Bad init is a vanishing/exploding gradient you created on purpose.
Why it matters
Before BN/LN, init was the difference between a trainable net and a brick. It still matters for small models and for the first few steps of large ones.
Case
A deep tanh net with N(0,1) weights saturates immediately. The same net with Xavier trains.
How to read it
Match init to activation (Xavier vs He). If you use LN/BN, init is less dramatic but not irrelevant.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Understanding the Difficulty of Training Deep Feedforward Neural NetworksGlorot, Bengio · 2010 · AISTATSXavier/Glorot init.