How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned

Abstract

Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low level sensor observations. Although a large portion of deep RL research has focused on applications in video games and simulated control, which does not connect with the constraints of learning in real environments, deep RL has also demonstrated promise in enabling physical robots to learn complex skills in the real world. At the same time,real world robotics provides an appealing domain for evaluating such algorithms, as it connects directly to how humans learn; as an embodied agent in the real world. Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains. In this review article, we present a number of case studies involving robotic deep RL. Building off of these case studies, we discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting and are not often the focus of mainstream RL research. Our goal is to provide a resource both for roboticists and machine learning researchers who are interested in furthering the progress of deep RL in the real world.

References in corpus (60)

• SMOTE: Synthetic Minority Over-sampling Technique
• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Learning agile and dynamic motor skills for legged robots
• End-to-End Training of Deep Visuomotor Policies
• Learning Quadrupedal Locomotion over Challenging Terrain
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
• Emergence of Locomotion Behaviours in Rich Environments
• OpenAI Gym
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Deep Exploration via Bootstrapped DQN
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Reinforcement Learning with Deep Energy-Based Policies
• Robust Adversarial Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Hindsight Experience Replay
• Multi-Task Learning as Multi-Objective Optimization
• Continuous Deep Q-Learning with Model-based Acceleration
• Off-Policy Deep Reinforcement Learning without Exploration
• One-Shot Visual Imitation Learning via Meta-Learning
• Model-Ensemble Trust-Region Policy Optimization
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• One-Shot Imitation Learning
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Variational Intrinsic Control
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Learning Symmetric and Low-energy Locomotion
• Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Constrained Policy Optimization
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Learning a visuomotor controller for real world robotic grasping using simulated depth images
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• An Optimistic Perspective on Offline Reinforcement Learning
• Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Residual Policy Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Learning to Walk via Deep Reinforcement Learning
• Policies Modulating Trajectory Generators
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• Task-Embedded Control Networks for Few-Shot Imitation Learning
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• Value constrained model-free continuous control
• Unsupervised Curricula for Visual Meta-Reinforcement Learning
• Learning to be Safe: Deep RL with a Safety Critic
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• Experience-Embedded Visual Foresight
• Learning Fast Adaptation with Meta Strategy Optimization

Cited by in corpus (45)

• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Run Time Assurance for Safety-Critical Systems: An Introduction to Safety Filtering Approaches for Complex Control Systems
• Active Learning in Robotics: A Review of Control Principles
• Autonomous Navigation for Robot-assisted Intraluminal and Endovascular Procedures: A Systematic Review
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• Robot Learning of Mobile Manipulation with Reachability Behavior Priors
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Reinforcement-learning-based actuator selection method for active flow control
• A Survey of the Potential Long-term Impacts of AI
• AI Challenges for Society and Ethics
• Maximum diffusion reinforcement learning
• Learning Implicit Priors for Motion Optimization
• Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective
• A roadmap for AI in robotics
• GraspLDM: Generative 6-DoF Grasp Synthesis using Latent Diffusion Models
• DeepCPG Policies for Robot Locomotion
• A Versatile Door Opening System with Mobile Manipulator through Adaptive Position-Force Control and Reinforcement Learning
• Planar Robot Casting with Real2Sim2Real Self-Supervised Learning
• Learning from Data Streams: An Overview and Update
• Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning
• A Safe Reinforcement Learning Algorithm for Supervisory Control of Power Plants
• CybORG: A Gym for the Development of Autonomous Cyber Agents
• TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality
• Real-Time Model-Free Deep Reinforcement Learning for Force Control of a Series Elastic Actuator
• Optimal Potential Shaping on SE(3) via Neural ODEs on Lie Groups
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler
• AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning
• Optimization-Based Algebraic Multigrid Coarsening Using Reinforcement Learning
• On exploration requirements for learning safety constraints
• Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot
• Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observability
• Learning Fast and Precise Pixel-to-Torque Control
• Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Reward
• What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks
• Optimal Gait Control for a Tendon-driven Soft Quadruped Robot by Model-based Reinforcement Learning
• Synthesizing Evolving Symbolic Representations for Autonomous Systems
• Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms
• Simulation-Aided Policy Tuning for Black-Box Robot Learning
• A Broad-persistent Advising Approach for Deep Interactive Reinforcement Learning in Robotic Environments
• Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories
• Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
• Koopman Spectrum Nonlinear Regulators and Efficient Online Learning
• SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning
• Dual Variational Autoencoders for Efficient Sim-to-Real Transfer in Low-Cost Robotic Navigation