UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons
arXiv:2309.07051 · doi:10.1145/3581783.3612503
Abstract
The automatic co-speech gesture generation draws much attention in computer animation. Previous works designed network structures on individual datasets, which resulted in a lack of data volume and generalizability across different motion capture standards. In addition, it is a challenging task due to the weak correlation between speech and gestures. To address these problems, we present UnifiedGesture, a novel diffusion model-based speech-driven gesture synthesis approach, trained on multiple gesture datasets with different skeletons. Specifically, we first present a retargeting network to learn latent homeomorphic graphs for different motion capture standards, unifying the representations of various gestures while extending the dataset. We then capture the correlation between speech and gestures based on a diffusion model architecture using cross-local attention and self-attention to generate better speech-matched and realistic gestures. To further align speech and gesture and increase diversity, we incorporate reinforcement learning on the discrete gesture units with a learned reward function. Extensive experiments show that UnifiedGesture outperforms recent approaches on speech-driven gesture generation in terms of CCA, FGD, and human-likeness. All code, pre-trained models, databases, and demos are available to the public at https://github.com/YoungSeng/UnifiedGesture.
arXiv comment16 pages, 11 figures, ACM MM 2023
References in corpus (30)
- Adam: A Method for Stochastic Optimization
- Decoupled Weight Decay Regularization
- Denoising Diffusion Probabilistic Models
- Training language models to follow instructions with human feedback
- Neural Discrete Representation Learning
- Soft Actor-Critic Algorithms and Applications
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
- Learning Quadrupedal Locomotion over Challenging Terrain
- Classifier-Free Diffusion Guidance
- Conservative Q-Learning for Offline Reinforcement Learning
- Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
- Deep Reinforcement Learning in Large Discrete Action Spaces
- Skeleton-Aware Networks for Deep Motion Retargeting
- Human Motion Diffusion Model
- Gesticulator: A framework for semantically-aware speech-driven gesture generation
- Modeling Human Motion with Quaternion-based Neural Networks
- Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
- The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation
- A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020
- AudioLM: a Language Modeling Approach to Audio Generation
- Generating coherent spontaneous speech and gesture from text
- The ReprGesture entry to the GENEA Challenge 2022
- Tuning computer vision models with task rewards
- Pretrained Diffusion Models for Unified Human Motion Synthesis
- Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
- EDGE: Editable Dance Generation From Music
- G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model
- FLAME: Free-form Language-based Motion Synthesis & Editing
- UDE: A Unified Driving Engine for Human Motion Generation
- Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Model