LoRA
LoRA fine-tunes a frozen model by learning low-rank adapters on selected weight matrices, cutting trainable parameters and storage versus full fine-tunes.
Definition
Hu et al. (2021) freeze W₀ and learn ΔW = BA with rank r ≪ d. Only A and B train. Merging BA into W₀ at serve time costs no extra latency. QLoRA later quantized the base to 4-bit so a single GPU can adapt a 65B model.
Why it matters
This is how most teams customize foundation models. Rank, target modules (QKV vs MLP), and data quality decide the result — not the brand of the base model alone.
Case
A 7B chat model + rank-16 LoRA on a desk FAQ becomes a useful assistant. The same LoRA on 200 leaked eval items “passes” the benchmark and fails every new ticket.
How to read it
Report rank, target modules, and whether adapters were merged. If you evaluate only the train domain, you measured memorization of the adapter set.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- LoRA: Low-Rank Adaptation of Large Language ModelsHu et al. · 2021 · ICLRThe LoRA paper.