Hierarchical Reinforcement Learning for Quadruped Locomotion

Abstract

Legged locomotion is a challenging task for learning algorithms, especially when the task requires a diverse set of primitive behaviors. To solve these problems, we introduce a hierarchical framework to automatically decompose complex locomotion tasks. A high-level policy issues commands in a latent space and also selects for how long the low-level policy will execute the latent command. Concurrently, the low-level policy uses the latent command and only the robot's on-board sensors to control the robot's actuators. Our approach allows the high-level policy to run at a lower frequency than the low-level one. We test our framework on a path-following task for a dynamic quadruped robot and we show that steering behaviors automatically emerge in the latent command space as low-level skills are needed for this task. We then show efficient adaptation of the trained policy to a different task by transfer of the trained low-level policy. Finally, we validate the policies on a real quadruped robot. To the best of our knowledge, this is the first application of end-to-end hierarchical learning to a real robotic locomotion task.

References in corpus (11)

• Learning agile and dynamic motor skills for legged robots
• Data-Efficient Hierarchical Reinforcement Learning
• Simple random search provides a competitive approach to reinforcement learning
• Meta Learning Shared Hierarchies
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Learning and Transfer of Modulated Locomotor Controllers
• Latent Space Policies for Hierarchical Reinforcement Learning
• Policy Transfer with Strategy Optimization
• Hierarchical Reinforcement Learning with Hindsight
• Policies Modulating Trajectory Generators
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Cited by in corpus (6)

• Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion
• Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning
• Gait Library Synthesis for Quadruped Robots via Augmented Random Search
• Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning
• How does the structure embedded in learning policy affect learning quadruped locomotion?
• Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks