Multi-Resolution Generative Modeling of Human Motion from Limited Data

Abstract

We present a generative model that learns to synthesize human motion from limited training sequences. Our framework provides conditional generation and blending across multiple temporal resolutions. The model adeptly captures human motion patterns by integrating skeletal convolution layers and a multi-scale architecture. Our model contains a set of generative and adversarial networks, along with embedding modules, each tailored for generating motions at specific frame rates while exerting control over their content and details. Notably, our approach also extends to the synthesis of co-speech gestures, demonstrating its ability to generate synchronized gestures from speech inputs, even with limited paired data. Through direct synthesis of SMPL pose parameters, our approach avoids test-time adjustments to fit human body meshes. Experimental results showcase our model's ability to achieve extensive coverage of training examples, while generating diverse motions, as indicated by local and global diversity metrics.

1O pages, 7 figures, published in European Conference on Visual Media Production CVMP 24

References in corpus (10)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• Skeleton-Aware Networks for Deep Motion Retargeting
• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• MoGlow: Probabilistic and controllable motion synthesis using normalising flows
• Gesticulator: A framework for semantically-aware speech-driven gesture generation
• Analyzing Input and Output Representations for Speech-Driven Gesture Generation
• GANimator: Neural Motion Synthesis from a Single Sequence
• Constructing Human Motion Manifold with Sequential Networks