7 papers
Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication
Filippo Merlo, Ece Takmaz, Wenkai Chen +1
To what degree and under what conditions do VLMs rely on scene context when generating references to objects? To address this question, we introduce the $\textit{Common Objects Out…
References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation
Silvia Casola, Yang Janet Liu, Siyao Peng +3
Human language production exhibits remarkable richness and variation, reflecting diverse communication styles and intents. However, this variation is often overlooked in summarizat…
VAQUUM: Are Vague Quantifiers Grounded in Visual Data?
Hugh Mee Wong, Rick Nouwen, Albert Gatt
Vague quantifiers such as "a few" and "many" are influenced by various contextual factors, including the number of objects present in a given context. In this work, we evaluate the…
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi +17
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reprodu…
FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamics
Yupei Du, Albert Gatt, Dong Nguyen
Despite the massive success of fine-tuning Pre-trained Language Models (PLMs), they remain susceptible to out-of-distribution input. Dataset cartography is a simple yet effective d…
A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences
Leonardo Bertolazzi, Albert Gatt, Raffaella Bernardi
The reasoning abilities of Large Language Models (LLMs) are becoming a central focus of study in NLP. In this paper, we consider the case of syllogistic reasoning, an area of deduc…