Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust

Faisal Mohammad, Mohamed Younis

Vision-language models (VLMs) are usually chosen from leaderboards, which give one ranking for every application. On questions about objects that a scene suggests but does not cont…