Residual Policy Learning

Abstract

We present Residual Policy Learning (RPL): a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch remains data-inefficient or intractable, but learning a residual on top of the initial controller can yield substantial improvements. We study RPL in six challenging MuJoCo tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration. For initial controllers, we consider both hand-designed policies and model-predictive controllers with known or learned transition models. By combining learning with control algorithms, RPL can perform long-horizon, sparse-reward tasks for which reinforcement learning alone fails. Moreover, we find that RPL consistently and substantially improves on the initial controllers. We argue that RPL is a promising approach for combining the complementary strengths of deep reinforcement learning and robotic control, pushing the boundaries of what either can achieve independently. Video and code at https://k-r-allen.github.io/residual-policy-learning/.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Continuous Deep Q-Learning with Model-based Acceleration
• Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing

Cited by in corpus (18)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Injecting Knowledge in Data-driven Vehicle Trajectory Predictors
• Evolving Rewards to Automate Reinforcement Learning
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Residual Reinforcement Learning from Demonstrations
• Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach
• Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning
• Grasp and Motion Planning for Dexterous Manipulation for the Real Robot Challenge
• Learning Vision-based Reactive Policies for Obstacle Avoidance
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer
• Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Bootstrapping Motor Skill Learning with Motion Planning
• Tolerance-Guided Policy Learning for Adaptable and Transferrable Delicate Industrial Insertion
• Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
• InsertionNet -- A Scalable Solution for Insertion
• Solving the Real Robot Challenge using Deep Reinforcement Learning