Learning Symmetric and Low-energy Locomotion

Abstract

Learning locomotion skills is a challenging problem. To generate realistic and smooth locomotion, existing methods use motion capture, finite state machines or morphology-specific knowledge to guide the motion generation algorithms. Deep reinforcement learning (DRL) is a promising approach for the automatic creation of locomotion control. Indeed, a standard benchmark for DRL is to automatically create a running controller for a biped character from a simple reward function. Although several different DRL algorithms can successfully create a running controller, the resulting motions usually look nothing like a real runner. This paper takes a minimalist learning approach to the locomotion problem, without the use of motion examples, finite state machines, or morphology-specific knowledge. We introduce two modifications to the DRL approach that, when used together, produce locomotion behaviors that are symmetric, low-energy, and much closer to that of a real person. First, we introduce a new term to the loss function (not the reward function) that encourages symmetric actions. Second, we introduce a new curriculum learning method that provides modulated physical assistance to help the character with left/right balance and forward movement. The algorithm automatically computes appropriate assistance to the character and gradually relaxes this assistance, so that eventually the character learns to move entirely without help. Because our method does not make use of motion capture data, it can be applied to a variety of character morphologies. We demonstrate locomotion controllers for the lower half of a biped, a full humanoid, a quadruped, and a hexapod. Our results show that learned policies are able to produce symmetric, low-energy gaits. In addition, speed-appropriate gait patterns emerge without any guidance from motion examples or contact planning.

Accepted to SIGGRAPH 2018. Supplementary video: https://www.youtube.com/watch?v=zkH90rU-uew&feature=youtu.be

References in corpus (1)

• Animating Human Athletics

Cited by in corpus (28)

• Learning agile and dynamic motor skills for legged robots
• Learning Quadrupedal Locomotion over Challenging Terrain
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• Embodied Intelligence via Learning and Evolution
• Character Controllers Using Motion VAEs
• ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters
• ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters
• Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation
• A Survey on Deep Learning for Skeleton-Based Human Animation
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitation
• Contact-Implicit Model Predictive Control: Controlling Diverse Quadruped Motions Without Pre-Planned Contact Modes or Trajectories
• AdaptNet: Policy Adaptation for Physics-Based Character Control
• Learning Whole-body Motor Skills for Humanoids
• Leveraging Symmetry in RL-based Legged Locomotion Control
• Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey
• Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions
• An Adaptable Approach to Learn Realistic Legged Locomotion without Examples
• Addressing imperfect symmetry: A novel symmetry-learning actor-critic extension
• Functionality-Driven Musculature Retargeting
• Towards Robust Direction Invariance in Character Animation
• A Survey on Reinforcement Learning Methods in Character Animation
• Discovering Fatigued Movements for Virtual Character Animation
• AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning
• Learning Human Search Behavior from Egocentric Visual Inputs
• Learning Human-like Locomotion Based on Biological Actuation and Rewards
• Deep Reinforcement Learning for High Level Character Control