Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

Samir Char, Carles Domingo-Enrich, Randall Balestriero

Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation for large-scale training, yet…