6 citations · 19 across the 22 of their papers we have counts for
25 papers
CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection
Pratham Gupta, Haotian Qi, Tushar Sandhan +1
We present CAST-ASD (Causal Attention over Speakers and Time), a causal joint model for active speaker detection (ASD). Recent ASD models improve performance by jointly processing…
Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection
Willem Berner, Julio Cesar Cavalcanti, Kalle Åström +1
Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues a…
MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild
Haotian Qi, Gabriel Skantze
Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduc…
Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?
Luca Modica, Filip Landin, Mehrdad Farahani +3
In recent years, several Speech Language Models (SLMs) that represent speech and written text jointly have been presented. The question then emerges about how model-internal mechan…
VoXtream2: Full-stream TTS with dynamic speaking rate control
Nikita Torgashov, Gustav Eje Henter, Gabriel Skantze
Full-stream text-to-speech (TTS) for interactive systems must start speaking with minimal delay while remaining controllable as text arrives incrementally. We present VoXtream2, a…
Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models
Yu Wang, Leyi Lao, Langchu Huang +3
Backchannels and fillers are important linguistic expressions in dialogue, but often treated as 'noise' to be bypassed in modern transformer-based language models (LMs). Here, we s…
Into the Wild: When Robots Are Not Welcome
Shaul Ashkenazi, Gabriel Skantze, Jane Stuart-Smith +1
Social robots are increasingly being deployed in public spaces, where they face not only technological difficulties and unexpected user utterances, but also objections from stakeho…
Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
Bram Willemsen, Gabriel Skantze
In this paper, we explore the use of a text-only, autoregressive language modeling approach for the extraction of referring expressions from visually grounded dialogue. More specif…
"Dyadosyncrasy", Idiosyncrasy and Demographic Factors in Turn-Taking
Julio Cesar Cavalcanti, Gabriel Skantze
Turn-taking in dialogue follows universal constraints but also varies significantly. This study examines how demographic (sex, age, education) and individual factors shape turn-tak…
Representation of perceived prosodic similarity of conversational feedback
Livia Qian, Carol Figueroa, Gabriel Skantze
Vocal feedback (e.g., `mhm', `yeah', `okay') is an important component of spoken dialogue and is crucial to ensuring common ground in conversational systems. The exact meaning of s…
Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection
Koji Inoue, Divesh Lala, Gabriel Skantze +1
In human conversations, short backchannel utterances such as "yeah" and "oh" play a crucial role in facilitating smooth and engaging dialogue. These backchannels signal attentivene…
Perception of Emotions in Human and Robot Faces: Is the Eye Region Enough?
Chinmaya Mishra, Gabriel Skantze, Peter Hagoort +1
The increased interest in developing next-gen social robots has raised questions about the factors affecting the perception of robot emotions. This study investigates the impact of…
Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guiding
Bram Willemsen, Gabriel Skantze
We propose an approach to referring expression generation (REG) in visually grounded dialogue that is meant to produce referring expressions (REs) that are both discriminative and…
Joint Learning of Context and Feedback Embeddings in Spoken Dialogue
Livia Qian, Gabriel Skantze
Short feedback responses, such as backchannels, play an important role in spoken dialogue. So far, most of the modeling of feedback responses has focused on their timing, often neg…
Multilingual Turn-taking Prediction Using Voice Activity Projection
Koji Inoue, Bing'er Jiang, Erik Ekstedt +2
This paper investigates the application of voice activity projection (VAP), a predictive turn-taking model for spoken dialogue, on multilingual data, encompassing English, Mandarin…
An Analysis of User Behaviors for Objectively Evaluating Spoken Dialogue Systems
Koji Inoue, Divesh Lala, Keiko Ochi +2
Establishing evaluation schemes for spoken dialogue systems is important, but it can also be challenging. While subjective evaluations are commonly used in user experiments, object…
Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection
Koji Inoue, Bing'er Jiang, Erik Ekstedt +2
A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialog…
Towards Objective Evaluation of Socially-Situated Conversational Robots: Assessing Human-Likeness through Multimodal User Behaviors
Koji Inoue, Divesh Lala, Keiko Ochi +2
This paper tackles the challenging task of evaluating socially situated conversational robots and presents a novel objective evaluation approach that relies on multimodal user beha…
Collecting Visually-Grounded Dialogue with A Game Of Sorts
Bram Willemsen, Dmytro Kalpakchi, Gabriel Skantze
An idealized, though simplistic, view of the referring expression production and grounding process in (situated) dialogue assumes that a speaker must merely appropriately specify t…
Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs
Agnes Axelsson, Gabriel Skantze
In any system that uses structured knowledge graph (KG) data as its underlying knowledge representation, KG-to-text generation is a useful tool for turning parts of the graph data…
Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis
Erik Ekstedt, Siyang Wang, Éva Székely +2
Turn-taking is a fundamental aspect of human communication where speakers convey their intention to either hold, or yield, their turn through prosodic cues. Using the recently prop…
What makes a good pause? Investigating the turn-holding effects of fillers
Bing'er Jiang, Erik Ekstedt, Gabriel Skantze
Filled pauses (or fillers), such as "uh" and "um", are frequent in spontaneous speech and can serve as a turn-holding cue for the listener, indicating that the current speaker is n…
Response-conditioned Turn-taking Prediction
Bing'er Jiang, Erik Ekstedt, Gabriel Skantze
Previous approaches to turn-taking and response generation in conversational systems have treated it as a two-stage process: First, the end of a turn is detected (based on conversa…
The Open-domain Paradox for Chatbots: Common Ground as the Basis for Human-like Dialogue
Gabriel Skantze, A. Seza Doğruöz
There is a surge in interest in the development of open-domain chatbots, driven by the recent advancements of large language models. The "openness" of the dialogue is expected to b…