35 citations · 68 across the 10 of their papers we have counts for
10 papers
Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning
Livia Qian, Gabriel Skantze
Backchannels (e.g., `yeah', `mhm', and `right') are short, non-interruptive feedback signals whose lexical form and prosody jointly convey pragmatic meaning. While prior computatio…
What Can You Say to a Robot? Capability Communication Leads to More Natural Conversations
Merle M. Reimann, Koen V. Hindriks, Florian A. Kunneman +3
When encountering a robot in the wild, it is not inherently clear to human users what the robot's capabilities are. When encountering misunderstandings or problems in spoken intera…
Resolving References in Visually-Grounded Dialogue via Text Generation
Bram Willemsen, Livia Qian, Gabriel Skantze
Vision-language models (VLMs) have shown to be effective at image retrieval based on simple text queries, but text-image retrieval based on conversational input remains a challenge…
How "open" are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation
A. Seza Doğruöz, Gabriel Skantze
Open-domain chatbots are supposed to converse freely with humans without being restricted to a topic, task or domain. However, the boundaries and/or contents of open-domain convers…
DBkWik++ -- Multi Source Matching of Knowledge Graphs
Sven Hertling, Heiko Paulheim
Large knowledge graphs like DBpedia and YAGO are always based on the same source, i.e., Wikipedia. But there are more wikis that contain information about long-tail entities such a…
Voice Activity Projection: Self-supervised Learning of Turn-taking Events
Erik Ekstedt, Gabriel Skantze
The modeling of turn-taking in dialog can be viewed as the modeling of the dynamics of voice activity of the interlocutors. We extend prior work and define the predictive task of V…
CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings
Gabriel Skantze, Bram Willemsen
This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained multimodal embedding model, where langua…
TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog
Erik Ekstedt, Gabriel Skantze
Syntactic and pragmatic completeness is known to be important for turn-taking prediction, but so far machine learning models of turn-taking have used such linguistic information in…
Multimodal Continuous Turn-Taking Prediction Using Multiscale RNNs
Matthew Roddy, Gabriel Skantze, Naomi Harte
In human conversational interactions, turn-taking exchanges can be coordinated using cues from multiple modalities. To design spoken dialog systems that can conduct fluid interacti…
Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs
Matthew Roddy, Gabriel Skantze, Naomi Harte
For spoken dialog systems to conduct fluid conversational interactions with users, the systems must be sensitive to turn-taking cues produced by a user. Models should be designed s…