Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech

Abstract

We propose a new framework for gesture generation, aiming to allow data-driven approaches to produce more semantically rich gestures. Our approach first predicts whether to gesture, followed by a prediction of the gesture properties. Those properties are then used as conditioning for a modern probabilistic gesture-generation model capable of high-quality output. This empowers the approach to generate gestures that are both diverse and representational. Follow-ups and more information can be found on the project page: https://svito-zar.github.io/speech2properties2gestures/ .

arXiv commentAccepted for publication at the ACM International Conference on Intelligent Virtual Agents (IVA 2021)

References in corpus (7)

• Normalizing Flows: An Introduction and Review of Current Methods
• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• MoGlow: Probabilistic and controllable motion synthesis using normalising flows
• Gesticulator: A framework for semantically-aware speech-driven gesture generation
• A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020
• Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation
• Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance

Cited by in corpus (4)

• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation
• Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis
• Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022