Learning to Walk in the Real World with Minimal Human Effort

Abstract

Reliable and stable locomotion has been one of the most fundamental challenges for legged robots. Deep reinforcement learning (deep RL) has emerged as a promising method for developing such control policies autonomously. In this paper, we develop a system for learning legged locomotion policies with deep RL in the real world with minimal human effort. The key difficulties for on-robot learning systems are automatic data collection and safety. We overcome these two challenges by developing a multi-task learning procedure and a safety-constrained RL framework. We tested our system on the task of learning to walk on three different terrains: flat ground, a soft mattress, and a doormat with crevices. Our system can automatically and efficiently learn locomotion skills on a Minitaur robot with little human intervention. The supplemental video can be found at: \url{https://youtu.be/cwyiq6dCgOc}.

References in corpus (23)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Learning agile and dynamic motor skills for legged robots
• Robots that can adapt like animals
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Emergence of Locomotion Behaviours in Rich Environments
• Solving Rubik's Cube with a Robot Hand
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Safe Exploration in Continuous Action Spaces
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Virtual to Real Reinforcement Learning for Autonomous Driving
• Variational End-to-End Navigation and Localization
• When to Trust Your Model: Model-Based Policy Optimization
• Reset-free Trial-and-Error Learning for Robot Damage Recovery
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Control
• Feedback Control For Cassie With Deep Reinforcement Learning
• Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation
• Learning Fast Adaptation with Meta Strategy Optimization
• Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning

Cited by in corpus (12)

• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Generalization in Transfer Learning
• Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation
• Observation Space Matters: Benchmark and Optimization Algorithm
• Automating Reinforcement Learning with Example-based Resets
• Smooth Exploration for Robotic Reinforcement Learning
• Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs
• Reinforcement Learning by Guided Safe Exploration
• Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning
• Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning