10 papers
Pathways of Visual Information Flow in Vision-Language Models
Israfel Salazar, Stella Frank, Dan Oneata +2
We study how visual information is routed in vision-language models (VLMs). Using causal patching on controlled synthetic and natural datasets, we find that models rely on two dist…
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
Mike Zhang, Ali Basirat, Desmond Elliott
Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in…
LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics
Galadrielle Humblot-Renaux, Mohammad N. S. Jahromi, Rohat Bakuri-Jørgensen +7
Off-the-shelf large language models (LLMs) are increasingly used to automate text annotation, yet their effectiveness remains underexplored for underrepresented languages and speci…
Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs
Israfel Salazar, Desmond Elliott, Yova Kementchedjhieva
Contrastive vision-language models (VLMs) have made significant progress in binding visual and textual information, yet understanding long, compositional captions remains an open c…
Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Danae Sánchez Villegas, Danae Sánchez Villegas, Samuel Lewis-Lim +2
Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear. We analyze reasoni…
Uncovering Cultural Representation Disparities in Vision-Language Models
Ram Mohan Rao Kadiyala, Siddhant Gupta, Jebish Purbey +4
Vision-Language Models (VLMs) have demonstrated impressive capabilities across a range of tasks, yet concerns about their potential biases exist. This work investigates the extent…