189 citations · 683 across the 22 of their papers we have counts for
22 papers
Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
Anna Deichler, Siyang Wang, Simon Alexanderson +1
One of the main goals of robotics and intelligent agent research is to enable natural communication with humans in physically situated settings. While recent work has focused on ve…
Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents
Anna Deichler, Simon Alexanderson, Jonas Beskow
This paper focuses on enhancing human-agent communication by integrating spatial context into virtual agents' non-verbal behaviors, specifically gestures. Recent advances in co-spe…
Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis
Shivam Mehta, Anna Deichler, Jim O'Regan +4
Although humans engaged in face-to-face conversation simultaneously communicate both verbally and non-verbally, methods for joint and unified synthesis of speech audio and co-speec…
Unified speech and gesture synthesis using flow matching
Shivam Mehta, Ruibo Tu, Simon Alexanderson +3
As text-to-speech technologies achieve remarkable naturalness in read-aloud tasks, there is growing interest in multimodal synthesis of verbal and non-verbal communicative behaviou…
Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation
Anna Deichler, Shivam Mehta, Simon Alexanderson +1
This paper describes a system developed for the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. Our solution builds on an existing dif…
Matcha-TTS: A fast TTS architecture with conditional flow matching
Shivam Mehta, Ruibo Tu, Jonas Beskow +2
We introduce Matcha-TTS, a new encoder-decoder architecture for speedy TTS acoustic modelling, trained using optimal-transport conditional flow matching (OT-CFM). This yields an OD…
Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis
Shivam Mehta, Siyang Wang, Simon Alexanderson +3
With read-aloud speech synthesis achieving high naturalness scores, there is a growing research interest in synthesising spontaneous speech. However, human spontaneous face-to-face…
Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
Simon Alexanderson, Rajmund Nagy, Jonas Beskow +1
Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models. We show that these models are an excellent fit for synthe…
OverFlow: Putting flows on top of neural transducers for better TTS
Shivam Mehta, Ambika Kirkland, Harm Lameris +3
Neural HMMs are a type of neural transducer recently proposed for sequence-to-sequence modelling in text-to-speech. They combine the best features of classic statistical speech syn…
Mechanical Chameleons: Evaluating the effects of a social robot's non-verbal behavior on social influence
Patrik Jonell, Anna Deichler, Ilaria Torre +2
In this paper we present a pilot study which investigates how non-verbal behavior affects social influence in social robots. We also present a modular system which is capable of co…
Integrated Speech and Gesture Synthesis
Siyang Wang, Simon Alexanderson, Joakim Gustafson +3
Text-to-speech and co-speech gesture synthesis have until now been treated as separate areas by two different research communities, and applications merely stack the two technologi…
Neural HMMs are all you need (for high-quality attention-free TTS)
Shivam Mehta, Éva Székely, Jonas Beskow +1
Neural sequence-to-sequence TTS has achieved significantly better output quality than statistical speech synthesis using HMMs. However, neural TTS is generally not probabilistic an…
Transflower: probabilistic autoregressive dance generation with multimodal attention
Guillermo Valle-Pérez, Gustav Eje Henter, Jonas Beskow +3
Dance requires skillful composition of complex movements that follow rhythmic, tonal and timbral features of music. Formally, generating dance conditioned on a piece of music can b…
Generating coherent spontaneous speech and gesture from text
Simon Alexanderson, Éva Székely, Gustav Eje Henter +2
Embodied human communication encompasses both verbal (speech) and non-verbal information (e.g., gesture and head movements). Recent advances in machine learning have substantially…
Can we trust online crowdworkers? Comparing online and offline participants in a preference test of virtual agents
Patrik Jonell, Taras Kucherenko, Ilaria Torre +1
Conducting user studies is a crucial component in many scientific fields. While some studies require participants to be physically present, other studies can be conducted both phys…
Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings
Patrik Jonell, Taras Kucherenko, Gustav Eje Henter +1
To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non…
MoGlow: Probabilistic and controllable motion synthesis using normalising flows
Gustav Eje Henter, Simon Alexanderson, Jonas Beskow
Data-driven modelling and synthesis of motion is an active research area with applications that include animation, games, and social robotics. This paper introduces a new class of…
The effect of a physical robot on vocabulary learning
Andreas Wedenborn, Preben Wik, Olov Engwall +1
This study investigates the effect of a physical robot taking the role of a teacher or exercise partner in a language learning exercise. In order to investigate this, an applicatio…
A Neural Network Approach to Missing Marker Reconstruction in Human Motion Capture
Taras Kucherenko, Jonas Beskow, Hedvig Kjellström
Optical motion capture systems have become a widely used technology in various fields, such as augmented reality, robotics, movie production, etc. Such systems use a large number o…
Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially-Aware Language Acquisition
Kalin Stefanov, Jonas Beskow, Giampiero Salvi
This paper presents a self-supervised method for visual detection of the active speaker in a multi-person spoken interaction scenario. Active speaker detection is a fundamental pre…
Machine Learning and Social Robotics for Detecting Early Signs of Dementia
Patrik Jonell, Joseph Mendelson, Thomas Storskog +12
This paper presents the EACare project, an ambitious multi-disciplinary collaboration with the aim to develop an embodied system, capable of carrying out neuropsychological tests t…
Visual Recognition of Isolated Swedish Sign Language Signs
Saad Akram, Jonas Beskow, Hedvig Kjellstrom
We present a method for recognition of isolated Swedish Sign Language signs. The method will be used in a game intended to help children training signing at home, as a complement t…