Self-Attention
Self-attention is attention where queries, keys, and values all come from the same sequence, so each position can mix information from every other position in one layer.
Definition
For a sequence X, we form Q = XW_Q, K = XW_K, V = XW_V and compute softmax(QKᵀ/√d)V. Multi-head splits this into several subspaces. Causal self-attention masks future positions for left-to-right language models.
Why it matters
This is the mixing primitive of GPT-style models. Cross-attention (decoder Q, encoder K/V) is the sibling used in encoder–decoder and in some multimodal stacks.
Case
A causal LM attending to the whole prefix can copy a rare entity mentioned 2k tokens ago. An RNN of similar size usually cannot. That long-range copy is why context windows became a product feature.
How to read it
Quadratic cost is real. If your sequence is 100k tokens, you need a memory-efficient kernel or a sparse pattern — not a bigger GPU wish.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Attention Is All You NeedVaswani et al. · 2017 · NeurIPSMulti-head self-attention definition.