System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

Tsan Tsai Chan, Varsha Suresh, Anisha Saha +2

Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text. However, existing mitigation str…