MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale

Abstract

General-purpose robotic systems must master a large repertoire of diverse skills to be useful in a range of daily tasks. While reinforcement learning provides a powerful framework for acquiring individual behaviors, the time needed to acquire each skill makes the prospect of a generalist robot trained with RL daunting. In this paper, we study how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks. In this framework new tasks can be continuously instantiated from previously learned tasks improving overall performance and capabilities of the system. To instantiate this system, we develop a scalable and intuitive framework for specifying new tasks through user-provided examples of desired outcomes, devise a multi-robot collective learning system for data collection that simultaneously collects experience for multiple tasks, and develop a scalable and generalizable multi-task deep reinforcement learning method, which we call MT-Opt. We demonstrate how MT-Opt can learn a wide range of skills, including semantic picking (i.e., picking an object from a particular category), placing into various fixtures (e.g., placing a food item onto a plate), covering, aligning, and rearranging. We train and evaluate our system on a set of 12 real-world tasks with data collected from 7 robots, and demonstrate the performance of our system both in terms of its ability to generalize to structurally similar new tasks, and acquire distinct new tasks more quickly by leveraging past experience. We recommend viewing the videos at https://karolhausman.github.io/mt-opt/

References in corpus (27)

• A Simple Neural Attentive Meta-Learner
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Solving Rubik's Cube with a Robot Hand
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Learning to reinforcement learn
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Multi-Task Learning as Multi-Objective Optimization
• Visual Reinforcement Learning with Imagined Goals
• One-Shot Visual Imitation Learning via Meta-Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Evolved Policy Gradients
• Deep Generative Dual Memory Network for Continual Learning
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Gradient Surgery for Multi-Task Learning
• Multi-Task Reinforcement Learning with Soft Modularization
• One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Meta-learning of Sequential Strategies
• Guided Meta-Policy Search
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight
• Multi-task Learning for Continuous Control

Cited by in corpus (15)

• On the Opportunities and Risks of Foundation Models
• CLIPort: What and Where Pathways for Robotic Manipulation
• Efficiently Identifying Task Groupings for Multi-Task Learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Learning Off-Policy with Online Planning
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
• Working Backwards: Learning to Place by Picking
• Autonomous Reinforcement Learning via Subgoal Curricula