Embedding
An embedding is a learned dense vector for an object (token, sentence, image, user) such that geometry supports retrieval, clustering, or as input to a downstream model.
Definition
Embeddings map discrete or high-dimensional objects into R^d. Token embeddings are a table; sentence embeddings pool a bidirectional encoder; document embeddings may add instruction prefixes (E5, GTE). Similarity is usually cosine or dot product after L2 norm.
Why it matters
This is the object ANN indexes store. RAG quality is embedding quality plus chunking. A beautiful chat model with a random embedding space will retrieve junk.
Case
A fine-tuned E5 on desk Q–A pairs lifts recall@10 from 0.4 to 0.8. The chat model did not change — the embedding space did. Conversely, indexing 8k-token chunks with a 512-token encoder silently truncates the evidence.
How to read it
Measure recall@k on a labeled query set from *this* corpus. MTEB leaderboard rank is not a substitute. Watch instruction prefixes: a document encoder and a query encoder may not be interchangeable.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Efficient Estimation of Word Representations in Vector SpaceMikolov, Chen, Corrado, Dean · 2013 · ICLR workshopIndustrial static embeddings.
- Text Embeddings by Weakly-Supervised Contrastive Pre-trainingWang et al. · 2022 · arXivE5 — a modern retrieval embedding recipe.