A Two-part Transformer Network for Controllable Motion Synthesis
arXiv:2304.12571 · doi:10.1109/TVCG.2023.3284402
Abstract
Although part-based motion synthesis networks have been investigated to reduce the complexity of modeling heterogeneous human motions, their computational cost remains prohibitive in interactive applications. To this end, we propose a novel two-part transformer network that aims to achieve high-quality, controllable motion synthesis results in real-time. Our network separates the skeleton into the upper and lower body parts, reducing the expensive cross-part fusion operations, and models the motions of each part separately through two streams of auto-regressive modules formed by multi-head attention layers. However, such a design might not sufficiently capture the correlations between the parts. We thus intentionally let the two parts share the features of the root joint and design a consistency loss to penalize the difference in the estimated root features and motions by these two auto-regressive modules, significantly improving the quality of synthesized motions. After training on our motion dataset, our network can synthesize a wide range of heterogeneous motions, like cartwheels and twists. Experimental and user study results demonstrate that our network is superior to state-of-the-art human motion synthesis networks in the quality of generated motions.
16 pages, 26 figures
References in corpus (16)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Decoupled Weight Decay Regularization
- WaveNet: A Generative Model for Raw Audio
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
- Character Controllers Using Motion VAEs
- Skeleton-Aware Networks for Deep Motion Retargeting
- MoGlow: Probabilistic and controllable motion synthesis using normalising flows
- VideoGPT: Video Generation using VQ-VAE and Transformers
- MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency
- Motion Puzzle: Arbitrary Motion Style Transfer by Body Part
- Learning to Generate Diverse Dance Motions with Transformer
- GANimator: Neural Motion Synthesis from a Single Sequence
- Transflower: probabilistic autoregressive dance generation with multimodal attention
- Real-time Controllable Motion Transition for Characters
- Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning