AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale

Abstract

Robotic skills can be learned via imitation learning (IL) using user-provided demonstrations, or via reinforcement learning (RL) using large amountsof autonomously collected experience.Both methods have complementarystrengths and weaknesses: RL can reach a high level of performance, but requiresexploration, which can be very time consuming and unsafe; IL does not requireexploration, but only learns skills that are as good as the provided demonstrations.Can a single method combine the strengths of both approaches? A number ofprior methods have aimed to address this question, proposing a variety of tech-niques that integrate elements of IL and RL. However, scaling up such methodsto complex robotic skills that integrate diverse offline data and generalize mean-ingfully to real-world scenarios still presents a major challenge. In this paper, ouraim is to test the scalability of prior IL + RL algorithms and devise a system basedon detailed empirical experimentation that combines existing components in themost effective and scalable way. To that end, we present a series of experimentsaimed at understanding the implications of each design decision, so as to develop acombined approach that can utilize demonstrations and heterogeneous prior datato attain the best performance on a range of real-world and realistic simulatedrobotic problems. Our complete method, which we call AW-Opt, combines ele-ments of advantage-weighted regression [1, 2] and QT-Opt [3], providing a unifiedapproach for integrating demonstrations and offline data for robotic manipulation.Please see https://awopt.github.io for more details.

References in corpus (16)

• Continuous control with deep reinforcement learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Soft Actor-Critic Algorithms and Applications
• Dota 2 with Large Scale Deep Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Conservative Q-Learning for Offline Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations
• RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real
• Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation