AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriors

Mohamed Eltahir, Fardows Adam, Duaa M. Tahir +6

When a VLM answers a visual query, current interpretability tools rely on text rationales, which use a mismatched modality, or on internal read-outs, which originate too early to r…