2 citations · 2 across the 11 of their papers we have counts for
12 papers
Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos
Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle
Foundation models are used for many real-world applications involving language understanding and generation from temporally-ordered multimodal events. In this work, we study the ab…
Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions
Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle
In recent years, a substantial body of work in visually grounded natural language processing has focused on real-life multimodal scenarios such as describing content depicted in im…
Vision-Language Models Align with Human Neural Representations in Concept Processing
Anna Bavaresco, Marianne de Heer Kloots, Sandro Pezzelle +1
Recent studies suggest that transformer-based vision-language models (VLMs) capture the multimodality of concept processing in the human brain. However, a systematic evaluation exp…
Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition
Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle
Visual storytelling consists in generating a natural language story given a temporally ordered sequence of images. This task is not only challenging for models, but also very diffi…
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi +17
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reprodu…
Describing Images : Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes
Ece Takmaz, Sandro Pezzelle, Raquel Fernández
There is an intricate relation between the properties of an image and how humans behave while describing the image. This behavior shows ample variation, as manifested in human sign…
GROOViST: A Metric for Grounding Objects in Visual Storytelling
Aditya K Surikuchi, Sandro Pezzelle, Raquel Fernández
A proper evaluation of stories generated for a sequence of images -- the task commonly referred to as visual storytelling -- must consider multiple aspects, such as coherence, gram…
The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models
Xinyi Chen, Raquel Fernández, Sandro Pezzelle
Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding…
Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind
Ece Takmaz, Nicolo' Brandizzi, Mario Giulianelli +2
Dialogue participants may have varying levels of knowledge about the topic under discussion. In such cases, it is essential for speakers to adapt their utterances by taking their a…
Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze
Ece Takmaz, Sandro Pezzelle, Lisa Beinborn +1
When speakers describe an image, they tend to look at objects before mentioning them. In this paper, we investigate such sequential cross-modal alignment by modelling the image des…
Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts
Ece Takmaz, Mario Giulianelli, Sandro Pezzelle +2
Dialogue participants often refer to entities or situations repeatedly within a conversation, which contributes to its cohesiveness. Subsequent references exploit the common ground…
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts
Sandro Pezzelle, Raquel Fernández
This work aims at modeling how the meaning of gradable adjectives of size (`big', `small') can be learned from visually-grounded contexts. Inspired by cognitive and linguistic evid…