COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning

Abstract

Reinforcement learning has been applied to a wide variety of robotics problems, but most of such applications involve collecting data from scratch for each new task. Since the amount of robot data we can collect for any single task is limited by time and cost considerations, the learned behavior is typically narrow: the policy can only execute the task in a handful of scenarios that it was trained on. What if there was a way to incorporate a large amount of prior data, either from previously solved tasks or from unsupervised or undirected environment interaction, to extend and generalize learned behaviors? While most prior work on extending robotic skills using pre-collected data focuses on building explicit hierarchies or skill decompositions, we show in this paper that we can reuse prior data to extend new skills simply through dynamic programming. We show that even when the prior data does not actually succeed at solving the new task, it can still be utilized for learning a better policy, by providing the agent with a broader understanding of the mechanics of its environment. We demonstrate the effectiveness of our approach by chaining together several behaviors seen in prior datasets for solving a new task, with our hardest experimental setting involving composing four robotic skills in a row: picking, placing, drawer opening, and grasping, where a +1/0 sparse reward is provided only on task completion. We train our policies in an end-to-end fashion, mapping high-dimensional image observations to low-level robot control commands, and present results in both simulated and real world domains. Additional materials and source code can be found on our project website: https://sites.google.com/view/cog-rl

arXiv commentAccepted to CoRL 2020. Source code and videos available at https://sites.google.com/view/cog-rl

References in corpus (20)

• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Learning Dexterous In-Hand Manipulation
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• An Optimistic Perspective on Offline Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• Learning Dexterous Manipulation Policies from Experience and Imitation
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• Interpretable Latent Spaces for Learning from Demonstration
• Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• Path Integral Guided Policy Search

Cited by in corpus (14)

• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• COMBO: Conservative Offline Model-Based Policy Optimization
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Demonstration-Guided Reinforcement Learning with Learned Skills
• Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization