11 papers
QUEST: A Query and Extraction System for Topics in Asylum Law Application Decisions
Maria Vlachou, Anna Murphy Høgenhaug, Mohammad N. S. Jahromi +4
Legal decisions on asylum applications consist of long, complex, and heterogeneous documents, covering narrative applicant interviews, original decisions, and additional supporting…
Pathways of Visual Information Flow in Vision-Language Models
Israfel Salazar, Stella Frank, Dan Oneata +2
We study how visual information is routed in vision-language models (VLMs). Using causal patching on controlled synthetic and natural datasets, we find that models rely on two dist…
LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics
Galadrielle Humblot-Renaux, Mohammad N. S. Jahromi, Rohat Bakuri-Jørgensen +7
Off-the-shelf large language models (LLMs) are increasingly used to automate text annotation, yet their effectiveness remains underexplored for underrepresented languages and speci…
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
Mike Zhang, Ali Basirat, Desmond Elliott
Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in…
Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras +1
Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear. We analyze reasoni…
Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs
Israfel Salazar, Desmond Elliott, Yova Kementchedjhieva
Contrastive vision-language models (VLMs) have made significant progress in binding visual and textual information, yet understanding long, compositional captions remains an open c…
How Well Do Vision-Language Models Understand Movies?
Emmanouil Zaranis, António Farinhas, Saul Santos +28
Vision-language models (VLMs) can increasingly handle extended contexts and answer questions about long videos--but can they understand the narrative structure of movies as humans…
Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era
Dan Oneata, Desmond Elliott, Stella Frank
Human learning and conceptual representation is grounded in sensorimotor experience, in contrast to state-of-the-art foundation models. In this paper, we investigate how well such…
Uncovering Cultural Representation Disparities in Vision-Language Models
Ram Mohan Rao Kadiyala, Siddhant Gupta, Jebish Purbey +4
Vision-Language Models (VLMs) have demonstrated impressive capabilities across a range of tasks, yet concerns about their potential biases exist. This work investigates the extent…
Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam +42
The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While mu…
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi +17
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reprodu…