GPT
GPT is a decoder-only Transformer trained to predict the next token. Scale plus this objective produced in-context learning and the current foundation-model product line.
Definition
Radford et al. train a causal Transformer on unlabeled text (GPT-1), show that larger scale plus more data yields task-agnostic learners (GPT-2), and that at sufficient scale the model does tasks from prompts alone (GPT-3). The likelihood is standard next-token cross-entropy.
Why it matters
This is the generative LM that the industry standardized on. Instruction tuning and RLHF sit on top of a GPT-style base; they do not replace the pretrain.
Case
GPT-3’s few-shot arithmetic and translation, without fine-tunes, is the empirical fact that created the prompt-engineering industry. It is also why people confuse “the model can complete text” with “the model knows.”
How to read it
Separate pretrain, instruction-tune, and alignment. A base GPT and a chat GPT are different artifacts. Report context length and decoding, or the anecdote is unusable.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Improving Language Understanding by Generative Pre-TrainingRadford, Narasimhan, Salimans, Sutskever · 2018 · OpenAIGPT-1.
- Language Models are Few-Shot LearnersBrown et al. · 2020 · NeurIPSGPT-3 and in-context learning at scale.