Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning

Abstract

Learning to locomote to arbitrary goals on hardware remains a challenging problem for reinforcement learning. In this paper, we present a hierarchical learning framework that improves sample-efficiency and generalizability of locomotion skills on real-world robots. Our approach divides the problem of goal-oriented locomotion into two sub-problems: learning diverse primitives skills, and using model-based planning to sequence these skills. We parametrize our primitives as cyclic movements, improving sample-efficiency of learning on a 18 degrees of freedom robot. Then, we learn coarse dynamics models over primitive cycles and use them in a model predictive control framework. This allows us to learn to walk to arbitrary goals up to 12m away, after about two hours of training from scratch on hardware. Our results on a Daisy hexapod hardware and simulation demonstrate the efficacy of our approach at reaching distant targets, in different environments and with sensory noise.

arXiv commentSubmitted to 2020 ICRA

References in corpus (11)

• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Learning agile and dynamic motor skills for legged robots
• Robots that can adapt like animals
• Emergence of Locomotion Behaviours in Rich Environments
• Hindsight Experience Replay
• Meta Learning Shared Hierarchies
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Learning Flexible and Reusable Locomotion Primitives for a Microrobot
• Curious iLQR: Resolving Uncertainty in Model-based RL
• Bayesian Optimization in Variational Latent Spaces with Dynamic Compression

Cited by in corpus (3)

• Learning to Walk in the Real World with Minimal Human Effort
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion