BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer

Abstract

Automatic gesture synthesis from speech is a topic that has attracted researchers for applications in remote communication, video games and Metaverse. Learning the mapping between speech and 3D full-body gestures is difficult due to the stochastic nature of the problem and the lack of a rich cross-modal dataset that is needed for training. In this paper, we propose a novel transformer-based framework for automatic 3D body gesture synthesis from speech. To learn the stochastic nature of the body gesture during speech, we propose a variational transformer to effectively model a probabilistic distribution over gestures, which can produce diverse gestures during inference. Furthermore, we introduce a mode positional embedding layer to capture the different motion speeds in different speaking modes. To cope with the scarcity of data, we design an intra-modal pre-training scheme that can learn the complex mapping between the speech and the 3D gesture from a limited amount of data. Our system is trained with either the Trinity speech-gesture dataset or the Talking With Hands 16.2M dataset. The results show that our system can produce more realistic, appropriate, and diverse body gestures compared to existing state-of-the-art approaches.

arXiv comment12 pages, 13 figures

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• MoGlow: Probabilistic and controllable motion synthesis using normalising flows
• Gesticulator: A framework for semantically-aware speech-driven gesture generation
• Analyzing Input and Output Representations for Speech-Driven Gesture Generation
• Audio to Body Dynamics
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020
• Integrated Speech and Gesture Synthesis

Cited by in corpus (2)

• Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis
• InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios