4 citations · 6 across the 10 of their papers we have counts for
12 papers
Investigating Multimodal Informativity under Different Partner Visibility Conditions in Video-Mediated Dialogue
Esam Ghaleb, Hugh Mee Wong, Kristina Kobrock
Situated language use is multimodal and embodied. For example, gestures can carry information that is absent or underspecified in the speech signal, yet dialogue models typically r…
Aligned but Not Partner-Specific: How Multimodal LLM Agents Succeed in Reference Games Without Forming Conceptual Pacts
Po-Ya Angela Wang, Chinmaya Mishra, Aslı Özyürek +2
Repeated reference games test whether interlocutors replace their initially long descriptions with shorter, partner-specific expressions grounded in shared interaction history; tha…
DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generation
Ferdinand Paar, Lanmiao Liu, Aslı Özyürek +2
Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic ges…
SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching
Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1
While the field of co-speech gesture generation has seen significant advances, producing holistic, semantically grounded gestures remains a challenge. Existing approaches rely on e…
The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping
Onur Keleş, Aslı Özyürek, Gerardo Ortega +2
Iconicity, the resemblance between linguistic form and meaning, is pervasive in signed languages, offering a natural testbed for visual grounding. For vision-language models (VLMs)…
SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning
Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1
Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating…
I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue
Esam Ghaleb, Bulat Khaertdinov, Aslı Özyürek +1
In face-to-face interaction, we use multiple modalities, including speech and gestures, to communicate information and resolve references to objects. However, how representational…
Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation
Esam Ghaleb, Bulat Khaertdinov, Wim Pouw +4
In face-to-face dialogues, the form-meaning relationship of co-speech gestures varies depending on contextual factors such as what the gestures refer to and the individual characte…
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi +17
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reprodu…
Analysing Cross-Speaker Convergence in Face-to-Face Dialogue through the Lens of Automatically Detected Shared Linguistic Constructions
Esam Ghaleb, Marlou Rasenberg, Wim Pouw +4
Conversation requires a substantial amount of coordination between dialogue participants, from managing turn taking to negotiating mutual understanding. Part of this coordination e…
Leveraging Speech for Gesture Detection in Multimodal Communication
Esam Ghaleb, Ilya Burenko, Marlou Rasenberg +7
Gestures are inherent to human interaction and often complement speech in face-to-face communication, forming a multimodal communication system. An important task in gesture analys…
Co-Speech Gesture Detection through Multi-Phase Sequence Labeling
Esam Ghaleb, Ilya Burenko, Marlou Rasenberg +6
Gestures are integral components of face-to-face communication. They unfold over time, often following predictable movement phases of preparation, stroke, and retraction. Yet, the…