In-Context Learning
In-context learning is when a frozen language model improves at a task from examples placed in the prompt, without weight updates.
Definition
Brown et al. showed that a large causal LM can infer a task from a few input–output pairs in the context window. No gradient step occurs. The mechanism is still debated (Bayesian-like inference over prompts vs pattern completion).
Why it matters
This is why prompting is a product surface. It is also why evals that leak the test format into the prompt overstate capability.
Case
GPT-3 translating from five demo pairs without a fine-tune is ICL. The same model asked with zero demos and a bad instruction will look “dumb.” That is prompt design, not a new architecture.
How to read it
Report number of shots, whether demos are from the test distribution, and decoding. ICL that requires the answer to be in the prompt is retrieval, not learning.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Language Models are Few-Shot LearnersBrown et al. · 2020 · NeurIPSICL at scale.