9 papers
Causal Attribution via Activation Patching
Amirmohammad Izadi, Mohammadali Banayeeanzade, Alireza Mirrokni +4
Attribution methods for Vision Transformers (ViTs) aim to identify image regions that influence model predictions, but producing faithful and well-localized attributions remains ch…
Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy
Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi +5
Large language models (LLMs), despite strong performance on complex mathematical problems, exhibit systematic limitations in counting tasks. This issue arises from the architectura…
Understanding Counting Mechanisms in Large Language and Vision-Language Models
Hosein Hasani, Amirmohammad Izadi, Fatemeh Askari +4
Counting is one of the fundamental abilities of large language models (LLMs) and large vision-language models (LVLMs). This paper examines how these foundation models represent and…
HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models
Reihaneh Zohrabi, Hosein Hasani, Akshita Gupta +3
Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly r…
Uncovering Grounding IDs: How External Cues Shape Multimodal Binding
Hosein Hasani, Amirmohammad Izadi, Fatemeh Askari +4
Large vision-language models (LVLMs) show strong performance across multimodal benchmarks but remain limited in structured reasoning and precise grounding. Recent work has demonstr…
Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs
Amirmohammad Izadi, Mohammad Ali Banayeeanzade, Fatemeh Askari +4
Despite progress in Large Vision-Language Models (LVLMs), their capacity for visual reasoning is often limited by the binding problem: the failure to reliably associate perceptual…