Hallucination
Hallucination is fluent generation that is not supported by the source or the world — a likelihood-trained model completing a pattern, not a database lookup.
Definition
In abstractive generation, hallucination means content unfaithful to the input (Maynez et al.). In open-domain QA it means claims not entailed by evidence. RLHF can make hallucinations *more* confident. The fix is retrieval, citations, abstention, or a tool — not a nicer system prompt alone.
Why it matters
This is the production risk that makes ungrounded chat unsafe for research notes, compliance, and trading commentary. Fluency is not a truth signal.
Case
A model cites “arXiv:2604.02302” because the number pattern is likely. The paper may not exist. RAG with clickable IDs is the control; prose confidence is not.
How to read it
Measure groundedness on a labeled set (supported / unsupported / abstain). If your eval only scores BLEU or “helpfulness,” you are selecting for better-sounding errors.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- On Faithfulness and Factuality in Abstractive SummarizationMaynez, Narayan, Bohnet, McDonald · 2020 · ACLIntrinsic vs extrinsic hallucination.