Tokenizer
A tokenizer splits raw text into the discrete tokens a model actually sees — bytes, characters, or learned subwords — and defines the vocabulary the softmax is over.
Definition
Neural sequence models do not read Unicode strings. A tokenizer maps text ↔ token IDs. Modern LLMs use subword tokenizers (BPE, WordPiece, Unigram/SentencePiece) so rare words decompose instead of becoming UNK. The tokenizer is part of the model: change it and every weight is wrong.
Why it matters
Tokenization errors show up as broken multilingual quality, weird numbers, and “the model cannot spell.” A 100k vocab that under-segments Chinese and over-segments code is a product bug, not an alignment bug.
Case
GPT-2’s BPE splits some numbers into arbitrary pieces, so arithmetic is harder than the parameter count suggests. Digit-friendly tokenizers later fixed a surprising fraction of “reasoning” failures.
How to read it
Look at fertility (tokens per word) on *your* corpus, not WikiText. If a contract ID becomes 12 tokens, retrieval and fine-tunes will be expensive and brittle.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Neural Machine Translation of Rare Words with Subword UnitsSennrich, Haddow, Birch · 2016 · ACLBPE subwords for NMT — the tokenizer most LLM stacks still descend from.
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text ProcessingKudo, Richardson · 2018 · EMNLPLanguage-independent Unigram/BPE used by T5, ALBERT, many LMs.