How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned

Abstract

Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low level sensor observations. Although a large portion of deep RL research has focused on applications in video games and simulated control, which does not connect with the constraints of learning in real environments, deep RL has also demonstrated promise in enabling physical robots to learn complex skills in the real world. At the same time,real world robotics provides an appealing domain for evaluating such algorithms, as it connects directly to how humans learn; as an embodied agent in the real world. Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains. In this review article, we present a number of case studies involving robotic deep RL. Building off of these case studies, we discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting and are not often the focus of mainstream RL research. Our goal is to provide a resource both for roboticists and machine learning researchers who are interested in furthering the progress of deep RL in the real world.

References in corpus (19)

• Learning agile and dynamic motor skills for legged robots
• Emergence of Locomotion Behaviours in Rich Environments
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Robust Adversarial Reinforcement Learning
• One-Shot Visual Imitation Learning via Meta-Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Variational Intrinsic Control
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Constrained Policy Optimization
• Learning a visuomotor controller for real world robotic grasping using simulated depth images
• Residual Policy Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Policies Modulating Trajectory Generators
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• Value constrained model-free continuous control
• Unsupervised Curricula for Visual Meta-Reinforcement Learning
• Learning to be Safe: Deep RL with a Safety Critic
• Experience-Embedded Visual Foresight

Cited by in corpus (10)

• Active Learning in Robotics: A Review of Control Principles
• Autonomous Navigation for Robot-assisted Intraluminal and Endovascular Procedures: A Systematic Review
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• DeepCPG Policies for Robot Locomotion
• CybORG: A Gym for the Development of Autonomous Cyber Agents
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• On exploration requirements for learning safety constraints
• Learning Fast and Precise Pixel-to-Torque Control
• What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks
• Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories