Byte Pair Encoding
BPE grows a vocabulary by repeatedly merging the most frequent adjacent pairs, starting from characters or bytes, until a target vocab size is reached.
Definition
Gage (1994) used BPE for compression. Sennrich et al. (2016) applied it to translation: start with characters, merge frequent pairs, encode unseen words as pieces. GPT-2/3-style tokenizers run BPE on bytes so the model never emits UNK.
Why it matters
BPE is the default LLM tokenizer algorithm. Fertility, multilingual fairness, and cache locality of the vocab are BPE design choices, not model-scale mysteries.
Case
A 32k BPE trained on English web text will shatter CJK and code. Retrain on a balanced mix and the same Transformer suddenly “learns” Chinese — you changed the tokenizer, not the intelligence.
How to read it
Inspect merge rules and a few encoded samples. If spaces, punctuation, or numbers look insane, fix BPE before you buy another GPU-week.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- A New Algorithm for Data CompressionGage · 1994 · C Users JournalOriginal BPE as compression.
- Neural Machine Translation of Rare Words with Subword UnitsSennrich, Haddow, Birch · 2016 · ACLBPE as a neural tokenizer.