Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis

Abstract

Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space. During training, the RFC-based policy learns to apply residual forces to the humanoid to compensate for the dynamics mismatch and better imitate the reference motion. Experiments on a wide range of dynamic motions demonstrate that our approach outperforms state-of-the-art methods in terms of convergence speed and the quality of learned motions. Notably, we showcase a physics-based virtual character empowered by RFC that can perform highly agile ballet dance moves such as pirouette, arabesque and jeté. Furthermore, we propose a dual-policy control framework, where a kinematic policy and an RFC-based policy work in tandem to synthesize multi-modal infinite-horizon human motions without any task guidance or user input. Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3.6M) and generates diverse long-term motions. Code and videos are available at https://www.ye-yuan.com/rfc.

arXiv commentNeurIPS 2020. Code: https://github.com/Khrylx/RFC. Project page: https://www.ye-yuan.com/rfc

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• QuaterNet: A Quaternion-based Recurrent Model for Human Motion
• Learning human behaviors from motion capture by adversarial imitation
• Diverse Trajectory Forecasting with Determinantal Point Processes
• Neural probabilistic motor primitives for humanoid control
• Hierarchical visuomotor control of humanoids
• DLow: Diversifying Latent Flows for Diverse Human Motion Prediction

Cited by in corpus (4)

• Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors
• C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters
• DLow: Diversifying Latent Flows for Diverse Human Motion Prediction
• Learning Control Policies for Imitating Human Gaits