Integrated Speech and Gesture Synthesis

Abstract

Text-to-speech and co-speech gesture synthesis have until now been treated as separate areas by two different research communities, and applications merely stack the two technologies using a simple system-level pipeline. This can lead to modeling inefficiencies and may introduce inconsistencies that limit the achievable naturalness. We propose to instead synthesize the two modalities in a single model, a new problem we call integrated speech and gesture synthesis (ISG). We also propose a set of models modified from state-of-the-art neural speech-synthesis engines to achieve this goal. We evaluate the models in three carefully-designed user studies, two of which evaluate the synthesized speech and gesture in isolation, plus a combined study that evaluates the models like they will be used in real-world applications -- speech and gesture presented together. The results show that participants rate one of the proposed integrated synthesis models as being as good as the state-of-the-art pipeline system we compare against, in all three tests. The model is able to achieve this with faster synthesis time and greatly reduced parameter count compared to the pipeline system, illustrating some of the potential benefits of treating speech and gesture synthesis together as a single, unified problem. Videos and code are available on our project page at https://swatsw.github.io/isg_icmi21/

9 pages, accepted at ICMI 2021

References in corpus (7)

• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• MoGlow: Probabilistic and controllable motion synthesis using normalising flows
• Gesticulator: A framework for semantically-aware speech-driven gesture generation
• A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020
• Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation
• HEMVIP: Human Evaluation of Multiple Videos in Parallel
• Generating coherent spontaneous speech and gesture from text

Cited by in corpus (5)

• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
• The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation
• Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022
• BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer