Teaching Vision-Language Models to Decline: Support Completion Against Mirage

Alfaj Uddin Ahmed, Md. Shaown Miah, Md Arid Hasan +2

Vision-language models answer questions about images they were never shown, a failure called mirage. Such an answer can be correct without being justified by the evidence, and the…