Word Embedding
A word embedding is a dense vector for a token such that geometry (distance, direction) reflects distributional meaning. It is the input layer of almost every neural NLP model.
Definition
Each token ID indexes a row of an embedding matrix E ∈ R^{V×d}. Training (language modeling, skip-gram, masked LM) moves those rows so similar contexts land nearby. Subword tokenizers still use embeddings — just on pieces, not words.
Why it matters
Embeddings are how discrete symbols enter continuous models. Retrieval, clustering, and “semantic search” are downstream uses of the same vectors (or of later contextual ones).
Case
Mikolov’s king − man + woman ≈ queen demo was a slide that funded a decade of vector semantics. Contextual models (BERT) later showed that a single static vector per word is too crude for polysemy.
How to read it
Static vs contextual matters. A word2vec table cannot disambiguate “bank.” A Transformer embedding + layers can. Do not mix them in one evaluation.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- A Neural Probabilistic Language ModelBengio, Ducharme, Vincent, Jauvin · 2003 · JMLRLearned distributed word vectors in a neural LM.
- Efficient Estimation of Word Representations in Vector SpaceMikolov, Chen, Corrado, Dean · 2013 · ICLR workshopword2vec.