Softmax
Softmax maps a real vector to a probability simplex: softmax(z)_i = exp(z_i) / Σ exp(z_j). It is the standard last layer for classification and attention weights.
Definition
Softmax turns logits into a categorical distribution. Temperature τ in softmax(z/τ) sharpens or flattens the mass. Numerically it is computed in log-space (log-softmax) to avoid overflow. Attention is a softmax over compatibility scores.
Why it matters
Every classifier and every Transformer attention head is a softmax. Temperature, masking, and label smoothing change calibration. A model that is “confident and wrong” is often a softmax + loss-design problem.
Case
Unmasked softmax over a 32k vocabulary will put tiny mass on every token and waste compute. Causal masking zeros illegal positions *before* softmax so the distribution stays on the legal set.
How to read it
Inspect the logits, not only the argmax. If max logit is huge, the model is overconfident; if logits are tiny and flat, it has not learned the task. Watch for NaNs when a row is all −∞ after masking.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern RecognitionBridle · 1990 · Neurocomputing (NATO ASI)Softmax as a normalized exponential class posterior.