Teacher Forcing
Teacher forcing trains a sequential decoder on the ground-truth previous token instead of its own prediction — fast and biased, which is why exposure bias exists.
Definition
During train, the decoder input at t is the true y_{t−1}. At test it is ŷ_{t−1}. That train/test mismatch is exposure bias. Scheduled sampling and professor forcing try to close it; modern LMs still teacher-force next-token likelihood.
Why it matters
It explains why a model that scores well on teacher-forced NLL can fall apart when it must live in its own samples.
Case
An early NMT model that is fluent for three words then degenerates: it never saw its own errors in train. Beam search hides some of this; sampling reveals it.
How to read it
If train NLL is great and free-run quality is not, you are looking at exposure bias (or a decoding bug). Not always ‘need a bigger model.’
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Scheduled Sampling for Sequence Prediction with Recurrent Neural NetworksBengio, Vinyals, Jaitly, Shazeer · 2015 · NeurIPSA standard attack on exposure bias.