Transformer
The Transformer is a sequence model built only from self-attention and feed-forward blocks, with no recurrence. It is the architecture behind BERT, GPT, T5, and almost every modern foundation model.
Definition
Vaswani et al. (2017) replace recurrence with multi-head self-attention, position encodings, residual connections, and layer norm. The encoder–decoder form was for translation; decoder-only (GPT) and encoder-only (BERT) are the two industrial forks.
Why it matters
This is the architecture. If you work on LLMs, retrieval, or modern vision, you are working on Transformer variants. Everything else in the 2017–2026 stack (scaling laws, RLHF, MoE, FlashAttention) assumes this block.
Case
“Attention Is All You Need” beat RNN NMT on WMT while training in a fraction of the wall-clock. The same block, scaled, is GPT-3. The jump was parallelism plus attention, not a new loss.
How to read it
Name the flavor (encoder / decoder / enc-dec), context length, and positional scheme (sinusoidal, RoPE, ALiBi). “We used a Transformer” is not a method section.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Attention Is All You NeedVaswani et al. · 2017 · NeurIPSThe Transformer paper.