Not All Attention Heads Are What You Need: Refining CLIP's Image Representation with Attention Ablation

Feng Lin, Marco Chen, Haokui Zhang +3

This paper investigates the role of attention heads in CLIP's image encoder. Building on interpretability studies, we conduct an exhaustive analysis and find that certain heads, di…