Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

Xurui Song, Weishi Wang, Zhongqi Yue +5

Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Langua…