Activation Function
An activation function is the pointwise nonlinearity between linear layers. Without it, stacked layers collapse to a single linear map.
Definition
Activations (σ, tanh, ReLU, GELU, SwiGLU) break linearity so a deep stack can represent non-linear functions. Choice affects gradient flow, sparsity, and smoothness. Transformers standardized on GELU / SwiGLU; vision nets still lean on ReLU variants.
Why it matters
Dead ReLUs, saturating sigmoids, and the wrong output activation (softmax vs linear) are silent model killers. The last-layer activation must match the loss: logits for cross-entropy, unbounded for regression.
Case
A price-forecast head with a sigmoid output cannot emit a return above 1. The net “trains,” metrics look finite, and the book is structurally capped. That is an activation bug, not a data bug.
How to read it
Check hidden-unit saturation and the output map against the target support. If most hidden units are zero or stuck at ±1, change the activation or the init before adding layers.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Deep Sparse Rectifier Neural NetworksGlorot, Bordes, Bengio · 2011 · AISTATSReLU as the default that made deep nets trainable.
- Gaussian Error Linear Units (GELUs)Hendrycks, Gimpel · 2016 · arXivThe smooth gate used in BERT and GPT-family models.