Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learning

Abstract

Deep reinforcement learning has emerged as a popular and powerful way to develop locomotion controllers for quadruped robots. Common approaches have largely focused on learning actions directly in joint space, or learning to modify and offset foot positions produced by trajectory generators. Both approaches typically require careful reward shaping and training for millions of time steps, and with trajectory generators introduce human bias into the resulting control policies. In this paper, we present a learning framework that leads to the natural emergence of fast and robust bounding policies for quadruped robots. The agent both selects and controls actions directly in task space to track desired velocity commands subject to environmental noise including model uncertainty and rough terrain. We observe that this framework improves sample efficiency, necessitates little reward shaping, leads to the emergence of natural gaits such as galloping and bounding, and eases the sim-to-real transfer at running speeds. Policies can be learned in only a few million time steps, even for challenging tasks of running over rough terrain with loads of over 100% of the nominal quadruped mass. Training occurs in PyBullet, and we perform a sim-to-sim transfer to Gazebo and sim-to-real transfer to the Unitree A1 hardware. For sim-to-sim, our results show the quadruped is able to run at over 4 m/s without a load, and 3.5 m/s with a 10 kg load, which is over 83% of the nominal quadruped mass. For sim-to-real, the Unitree A1 is able to bound at 2 m/s with a 5 kg load, representing 42% of the nominal quadruped mass.

References in corpus (10)

• Learning agile and dynamic motor skills for legged robots
• Learning Quadrupedal Locomotion over Challenging Terrain
• Learning robust perceptive locomotion for quadrupedal robots in the wild
• Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion
• Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
• Learning Free Gait Transition for Quadruped Robots via Phase-Guided Controller
• Policies Modulating Trajectory Generators
• Robust Quadruped Jumping via Deep Reinforcement Learning
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• Dynamics and Domain Randomized Gait Modulation with Bezier Curves for Sim-to-Real Legged Locomotion

Cited by in corpus (4)

• Adaptive Force-Based Control of Dynamic Legged Locomotion over Uneven Terrain
• Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds
• Towards Torque-Driven Reinforcement Learning for Quadruped Locomotion
• Advances, challenges, and opportunities for legged robots