Attention Sinks in VLMs Are Borrowed, Not Seen

Dengjia Zhang, Alexander Martin, Pengyu Yan +6

We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a large share of attention over the…