Positional Encoding
Positional encodings inject order into a permutation-invariant attention mixer so the model knows that token i is not token j.
Definition
Pure attention has no built-in order. Vaswani et al. added sinusoidal encodings; later models use learned embeddings, RoPE (rotary), or ALiBi (linear biases). The choice governs length extrapolation.
Why it matters
A model that cannot tell “dog bites man” from “man bites dog” is not a language model. Position is also why “we trained at 2k and served at 32k” is a research problem, not a config flag.
Case
RoPE in LLaMA-style models lets you stretch context with interpolation or NTK-aware scaling. Sinusoidal absolute encodings usually die outside the train length.
How to read it
Ask how positions are represented and what happens past train length. If the paper is silent, assume the model was never tested there.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Attention Is All You NeedVaswani et al. · 2017 · NeurIPSSinusoidal positional encodings.
- RoFormer: Enhanced Transformer with Rotary Position EmbeddingSu et al. · 2021 · NeurocomputingRoPE.