Knowledge Distillation
Knowledge distillation trains a smaller student to match a teacher’s output distribution (soft labels), transferring behavior without copying every weight.
Definition
Hinton et al.: minimize KL(teacher ‖ student) on softened logits (temperature T > 1) plus, optionally, hard-label loss. The soft targets carry ‘dark knowledge’ about similarities among classes. Sequence-level and feature-level variants exist for LMs.
Why it matters
It is how you ship a 7B that behaves like a 70B on a narrow task, and how some labs train without publishing the teacher.
Case
A distilled classifier on a desk’s five labels can beat a larger general model that was never shown those labels. Distilling a hallucinated teacher just distills the hallucination.
How to read it
Report teacher, temperature, and the student’s independent eval. If the student only matches teacher errors, you compressed the bug.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Distilling the Knowledge in a Neural NetworkHinton, Vinyals, Dean · 2015 · NeurIPS workshopThe distillation paper.