Character Controllers Using Motion VAEs

Abstract

A fundamental problem in computer animation is that of realizing purposeful and realistic human movement given a sufficiently-rich set of motion capture clips. We learn data-driven generative models of human movement using autoregressive conditional variational autoencoders, or Motion VAEs. The latent variables of the learned autoencoder define the action space for the movement and thereby govern its evolution over time. Planning or control algorithms can then use this action space to generate desired motions. In particular, we use deep reinforcement learning to learn controllers that achieve goal-directed movements. We demonstrate the effectiveness of the approach on multiple tasks. We further evaluate system-design choices and describe the current limitations of Motion VAEs.

Project page: https://www.cs.ubc.ca/~hyuling/projects/mvae/ ; Code: https://github.com/electronicarts/character-motion-vaes

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Emergence of Locomotion Behaviours in Rich Environments
• OpenAI Gym
• Go-Explore: a New Approach for Hard-Exploration Problems
• Learning Symmetric and Low-energy Locomotion
• Meta Learning Shared Hierarchies
• Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation

Cited by in corpus (57)

• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters
• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• MoGlow: Probabilistic and controllable motion synthesis using normalising flows
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
• A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
• ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• A Survey on Deep Learning for Skeleton-Based Human Animation
• LoBSTr: Real-time Lower-body Pose Prediction from Sparse Upper-body Tracking Signals
• CALM: Conditional Adversarial Latent Models for Directable Virtual Characters
• Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
• Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure
• Real-time Controllable Motion Transition for Characters
• Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors
• Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
• Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras
• PADL: Language-Directed Physics-Based Character Control
• Conditional Motion In-betweening
• Composite Motion Learning with Task Control
• A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control
• Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model
• AdaptNet: Policy Adaptation for Physics-Based Character Control
• RSMT: Real-time Stylized Motion Transition for Characters
• Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras with Human Semantics
• Pose Representations for Deep Skeletal Animation
• RigAnything: Template-Free Autoregressive Rigging for Diverse 3D Assets
• Placing Human Animations into 3D Scenes by Learning Interaction- and Geometry-Driven Keyframes
• MOVIN: Real-time Motion Capture using a Single LiDAR
• Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
• A causal convolutional neural network for multi-subject motion modeling and generation
• Decoupling Contact for Fine-Grained Motion Style Transfer
• We are More than Our Joints: Predicting how 3D Bodies Move
• Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
• A Two-part Transformer Network for Controllable Motion Synthesis
• Generating Smooth Pose Sequences for Diverse Human Motion Prediction
• Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders
• Adaptive Tracking of a Single-Rigid-Body Character in Various Environments
• SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation
• A Survey on Reinforcement Learning Methods in Character Animation
• PACE: Data-Driven Virtual Agent Interaction in Dense and Cluttered Environments
• Environment-aware Motion Matching
• ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling
• Task-Generic Hierarchical Human Motion Prior using VAEs
• DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization
• 3HANDS Dataset: Learning from Humans for Generating Naturalistic Handovers with Supernumerary Robotic Limbs
• Real-time Diverse Motion In-betweening with Space-time Control
• Learning to Ball: Composing Policies for Long-Horizon Basketball Moves
• Learning Calibratable Policies using Programmatic Style-Consistency
• Exploring Versatile Prior for Human Motion via Motion Frequency Guidance
• Discovering Diverse Athletic Jumping Strategies
• Deep Generative Modelling of Human Reach-and-Place Action
• ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation
• Hierarchical Graph-Convolutional Variational AutoEncoding for Generative Modelling of Human Motion
• STyMo: Fast and Controllable Few-Shot Motion Style Transfer
• Deep Reinforcement Learning for High Level Character Control