CLIP
CLIP jointly trains an image encoder and a text encoder so matched image–caption pairs are close in a shared space, enabling zero-shot visual classification by text prompts.
Definition
Radford et al. (2021) contrastively train dual encoders on 400M image–text pairs. At test time, class names become text embeddings; the image is assigned the nearest one. No task-specific labels are required for the zero-shot head.
Why it matters
CLIP is the default vision–language embedding. Retrieval, filters, and many multimodal RAGs start here. It also inherits web-crawl biases — “zero-shot” is not “unbiased.”
Case
Prompting CLIP with “a photo of a {class}” beat many supervised ImageNet linear probes. Prompting it with finance-chart slang it never saw does nothing useful.
How to read it
Zero-shot accuracy is prompt-dependent. Report the prompt set. For retrieval, measure recall@k on *your* image–text pairs, not ImageNet folklore.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Learning Transferable Visual Models From Natural Language SupervisionRadford et al. · 2021 · ICMLCLIP.