BERT
BERT is a bidirectional Transformer encoder trained with masked language modeling and next-sentence prediction, then fine-tuned on downstream NLP tasks.
Definition
Devlin et al. (2018) pretrain a Transformer encoder by hiding tokens and asking the model to fill them using left *and* right context. Fine-tuning a classification head then set a new SOTA on GLUE. It is not a left-to-right generator.
Why it matters
BERT is why “pretrained encoder + small head” became the default NLP stack. Sentence embeddings, token classification, and many retrieval encoders still live in this family (RoBERTa, E5, BERT-style rerankers).
Case
A legal NER tagger trained from scratch on 5k docs loses to BERT fine-tuned for one epoch. The win is the pretrained contextual encoder, not the CRF you bolted on.
How to read it
BERT cannot be used as a GPT-style chatbot without a different objective. If you need generation, use a decoder. If you need bidirectional encoding, BERT-family is still correct.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingDevlin, Chang, Lee, Toutanova · 2019 · NAACLThe BERT paper.