Attention Mechanism
Attention builds a weighted average of values, with weights from a compatibility function of queries and keys. It lets a model focus on relevant parts of a context instead of a single fixed vector.
Definition
Bahdanau attention (2014) let a decoder look at every encoder state, weighted by a learned alignment. Luong et al. simplified the score. Transformers made attention the *only* sequence mixer: softmax(QKᵀ/√d) V.
Why it matters
Attention is the reason sequence models can use long context without a single bottleneck vector. It is also quadratic in sequence length — the cost that FlashAttention and sparse variants exist to pay.
Case
Early NMT without attention had to squash a 50-word sentence into one vector and forgot the source. With attention, alignments became inspectable and BLEU jumped.
How to read it
Look at alignment or attention maps when the model fails. Uniform attention means the head is not selecting; attention on BOS/padding means the mask is wrong.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau, Cho, Bengio · 2015 · ICLRAdditive attention — the first widely used align-and-translate model.
- Effective Approaches to Attention-based Neural Machine TranslationLuong, Pham, Manning · 2015 · EMNLPDot-product and local attention variants.