Neural probabilistic motor primitives for humanoid control

Abstract

We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning. We encourage readers to view a supplementary video ( https://youtu.be/CaDEf-QcKwA ) summarizing our results.

arXiv commentAccepted as a conference paper at ICLR 2019

Cited by in corpus (34)

• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• dm_control: Software and Tasks for Continuous Control
• Critic Regularized Regression
• A Survey on Deep Learning for Skeleton-Based Human Animation
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie
• Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
• C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis
• Composite Motion Learning with Task Control
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Behavior Priors for Efficient Reinforcement Learning
• UniCon: Universal Neural Controller For Physics-based Character Motion
• Learning to Brachiate via Simplified Model Imitation
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Deep neuroethology of a virtual rodent
• Demonstration-Guided Reinforcement Learning with Learned Skills
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control
• Augmenting learning using symmetry in a biologically-inspired domain
• Product Kanerva Machines: Factorized Bayesian Memory
• Thalamocortical motor circuit insights for more robust hierarchical control of complex sequences
• REALab: An Embedded Perspective on Tampering
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval
• SimPoE: Simulated Character Control for 3D Human Pose Estimation
• Hierarchical Skills for Efficient Exploration
• Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
• Learning the Solution Manifold in Optimization and Its Application in Motion Planning
• Manipulator-Independent Representations for Visual Imitation
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• Neural Embedding for Physical Manipulations