5 papers · 1 filter
Don't Learn, Ground: A Case for Natural Language Inference with Visual Grounding
Daniil Ignatev, Ayman Santeer, Albert Gatt +1
We propose a zero-shot method for Natural Language Inference (NLI) that leverages multimodal representations by grounding language in visual contexts. Our approach generates visual…
From Image Captioning to Visual Storytelling
Admitos Passadakis, Yingjin Song, Albert Gatt
Visual Storytelling is a challenging multimodal task between Vision & Language, where the purpose is to generate a story for a stream of images. Its difficulty lies on the fact tha…
Disentangling the Roles of Representation and Selection in Data Pruning
Yupei Du, Yingjin Song, Hugh Mee Wong +3
Data pruning, selecting small but impactful subsets, offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design c…
Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?
Yingjin Song, Yupei Du, Denis Paperno +1
This paper introduces the TempVS benchmark, which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models (MLLMs) in image sequences. TempVS co…
Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning
Yingjin Song, Denis Paperno, Albert Gatt
Visual storytelling systems generate multi-sentence stories from image sequences. In this task, capturing contextual information and bridging visual variation bring additional chal…