Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Abstract

Designing agile locomotion for quadruped robots often requires extensive expertise and tedious manual tuning. In this paper, we present a system to automate this process by leveraging deep reinforcement learning techniques. Our system can learn quadruped locomotion from scratch using simple reward signals. In addition, users can provide an open loop reference to guide the learning process when more control over the learned gait is needed. The control policies are learned in a physics simulator and then deployed on real robots. In robotics, policies trained in simulation often do not transfer to the real world. We narrow this reality gap by improving the physics simulator and learning robust policies. We improve the simulation using system identification, developing an accurate actuator model and simulating latency. We learn robust controllers by randomizing the physical environments, adding perturbations and designing a compact observation space. We evaluate our system on two agile locomotion gaits: trotting and galloping. After learning in simulation, a quadruped robot can successfully perform both gaits in the real world.

Accompanying video: https://www.youtube.com/watch?v=lUZUr7jxoqM

References in corpus (9)

• Continuous control with deep reinforcement learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Emergence of Locomotion Behaviours in Rich Environments
• DeepMind Control Suite
• A Terradynamics of Legged Locomotion on Granular Media
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• The Penn Jerboa: A Platform for Exploring Parallel Composition of Templates
• TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow

Cited by in corpus (46)

• Learning agile and dynamic motor skills for legged robots
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion
• Multimodal Attention Branch Network for Perspective-Free Sentence Generation
• Self-driving scale car trained by Deep reinforcement learning
• Robotic Table Tennis: A Case Study into a High Speed Learning System
• Adaptive Power System Emergency Control using Deep Reinforcement Learning
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model
• Hierarchical Reinforcement Learning for Quadruped Locomotion
• MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics
• Fast and Efficient Locomotion via Learned Gait Transitions
• Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs
• Evaluating the Robustness of Collaborative Agents
• Reinforcement Learning with Adaptive Curriculum Dynamics Randomization for Fault-Tolerant Robot Control
• Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion
• ADAIL: Adaptive Adversarial Imitation Learning
• Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Learning to Manipulate Object Collections Using Grounded State Representations
• Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning
• Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS Biped
• qiBullet, a Bullet-based simulator for the Pepper and NAO robots
• Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
• Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion
• Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning
• Learning Compositional Neural Programs for Continuous Control
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Deep Learned Path Planning via Randomized Reward-Linked-Goals and Potential Space Applications
• Deep Reinforcement Learning with Linear Quadratic Regulator Regions
• Multi-Fidelity Reinforcement Learning with Gaussian Processes
• GEM: Group Enhanced Model for Learning Dynamical Control Systems
• Online Algorithms and Policies Using Adaptive and Machine Learning Approaches
• Weighted Entropy Modification for Soft Actor-Critic
• Learning to See before Learning to Act: Visual Pre-training for Manipulation
• Sim-to-Real Transfer of Robot Learning with Variable Length Inputs
• Similarity metrics for Different Market Scenarios in Abides
• PAMTRI: Pose-Aware Multi-Task Learning for Vehicle Re-Identification Using Highly Randomized Synthetic Data
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents