Robust Motion In-betweening

Abstract

In this work we present a novel, robust transition generation technique that can serve as a new tool for 3D animators, based on adversarial recurrent neural networks. The system synthesizes high-quality motions that use temporally-sparse keyframes as animation constraints. This is reminiscent of the job of in-betweening in traditional animation pipelines, in which an animator draws motion frames between provided keyframes. We first show that a state-of-the-art motion prediction model cannot be easily converted into a robust transition generator when only adding conditioning information about future keyframes. To solve this problem, we then propose two novel additive embedding modifiers that are applied at each timestep to latent representations encoded inside the network's architecture. One modifier is a time-to-arrival embedding that allows variations of the transition length with a single model. The other is a scheduled target noise vector that allows the system to be robust to target distortions and to sample different transitions given fixed keyframes. To qualitatively evaluate our method, we present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios. To quantitatively evaluate performance on transitions and generalizations to longer time horizons, we present well-defined in-betweening benchmarks on a subset of the widely used Human3.6M dataset and on LaFAN1, a novel high quality motion capture dataset that is more appropriate for transition generation. We are releasing this new dataset along with this work, with accompanying code for reproducing our baseline results.

Published at SIGGRAPH 2020

References in corpus (9)

• On the Convergence of Adam and Beyond
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• A Learned Representation For Artistic Style
• Generative Adversarial Imitation Learning
• FiLM: Visual Reasoning with a General Conditioning Layer
• Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamic
• Model-based Adversarial Imitation Learning
• PLU: The Piecewise Linear Unit Activation Function
• Learning Human Motion Models for Long-term Predictions

Cited by in corpus (30)

• MoGlow: Probabilistic and controllable motion synthesis using normalising flows
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
• InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions
• Convolutional Autoencoders for Human Motion Infilling
• ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters
• GANimator: Neural Motion Synthesis from a Single Sequence
• A Survey on Deep Learning for Skeleton-Based Human Animation
• Real-time Controllable Motion Transition for Characters
• C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters
• Single-Shot Motion Completion with Transformer
• Conditional Motion In-betweening
• A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control
• Composite Motion Learning with Task Control
• AdaptNet: Policy Adaptation for Physics-Based Character Control
• RSMT: Real-time Stylized Motion Transition for Characters
• Motion In-Betweening with Phase Manifolds
• Scene Synthesis from Human Motion
• Pose-aware Attention Network for Flexible Motion Retargeting by Body Part
• Pose Representations for Deep Skeletal Animation
• Placing Human Animations into 3D Scenes by Learning Interaction- and Geometry-Driven Keyframes
• Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation
• Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
• Deep Residual Mixture Models
• Progressive Human Motion Generation Based on Text and Few Motion Frames
• Task-Generic Hierarchical Human Motion Prior using VAEs
• PACE: Data-Driven Virtual Agent Interaction in Dense and Cluttered Environments
• Real-time Diverse Motion In-betweening with Space-time Control
• Motion Keyframe Interpolation for Any Human Skeleton via Temporally Consistent Point Cloud Sampling and Reconstruction
• GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras