Fine-Tuning
Fine-tuning continues training a pretrained model on a narrower distribution so the same weights specialize — classification heads, instruction following, or a desk domain.
Definition
Pretrain on a broad corpus, then update all or some weights on task data. ULMFiT and BERT made this the default NLP recipe; instruction SFT is the LM version. Catastrophic forgetting is the failure mode: the model gains the task and loses the base.
Why it matters
You almost never want to train a foundation model from scratch. Fine-tuning (or LoRA) is how Codex-like products specialize. Data quality beats learning-rate folklore.
Case
BERT + one epoch on a labeled desk corpus beats a from-scratch BiLSTM on the same labels. A full-weight fine-tune on 80 chat transcripts will also destroy the base model’s general English.
How to read it
State what is frozen, the LR vs pretrain LR, and a general-ability holdout. If you only measure the fine-tune task, you cannot see forgetting.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Universal Language Model Fine-tuning for Text ClassificationHoward, Ruder · 2018 · ACLULMFiT — practical transfer for NLP.
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingDevlin et al. · 2019 · NAACLPretrain then fine-tune as the industrial recipe.