Learning to Walk via Deep Reinforcement Learning

Abstract

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website.

RSS 2019, https://sites.google.com/view/minitaur-locomotion/

Cited by in corpus (22)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Data Efficient Reinforcement Learning for Legged Robots
• Learning to Fly via Deep Model-Based Reinforcement Learning
• Evolving Rewards to Automate Reinforcement Learning
• RMA: Rapid Motor Adaptation for Legged Robots
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Asynchronous Methods for Model-Based Reinforcement Learning
• Observation Space Matters: Benchmark and Optimization Algorithm
• Bayesian Optimization in Variational Latent Spaces with Dynamic Compression
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
• Zero-Shot Terrain Generalization for Visual Locomotion Policies
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• Combining Benefits from Trajectory Optimization and Deep Reinforcement Learning
• Human-Robot Collaboration via Deep Reinforcement Learning of Real-World Interactions
• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Towards Embodied Scene Description
• OTTR: Off-Road Trajectory Tracking using Reinforcement Learning
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• Learning Agile Locomotion Skills with a Mentor
• Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime
• Quadruped Locomotion on Non-Rigid Terrain using Reinforcement Learning