126 citations · 161 across the 36 of their papers we have counts for
7 papers · 1 filter
What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?
Anna Bavaresco, Ina Klarić, Raquel Fernández +1
Image descriptions represented with language models (LMs) predict human brain responses to naturalistic images in high-level visual regions, but the factors driving this predictivi…
Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos
Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle
Foundation models are used for many real-world applications involving language understanding and generation from temporally-ordered multimodal events. In this work, we study the ab…
How Well Do Vision-Language Models Understand Movies?
Emmanouil Zaranis, António Farinhas, Saul Santos +28
Vision-language models (VLMs) can increasingly handle extended contexts and answer questions about long videos--but can they understand the narrative structure of movies as humans…
I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue
Esam Ghaleb, Bulat Khaertdinov, Aslı Özyürek +1
In face-to-face interaction, we use multiple modalities, including speech and gestures, to communicate information and resolve references to objects. However, how representational…
Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation
Esam Ghaleb, Bulat Khaertdinov, Wim Pouw +4
In face-to-face dialogues, the form-meaning relationship of co-speech gestures varies depending on contextual factors such as what the gestures refer to and the individual characte…
Leveraging Speech for Gesture Detection in Multimodal Communication
Esam Ghaleb, Ilya Burenko, Marlou Rasenberg +7
Gestures are inherent to human interaction and often complement speech in face-to-face communication, forming a multimodal communication system. An important task in gesture analys…
Co-Speech Gesture Detection through Multi-Phase Sequence Labeling
Esam Ghaleb, Ilya Burenko, Marlou Rasenberg +6
Gestures are integral components of face-to-face communication. They unfold over time, often following predictable movement phases of preparation, stroke, and retraction. Yet, the…