Multi-Task Reinforcement Learning with Context-based Representations

Abstract

The benefit of multi-task learning over single-task learning relies on the ability to use relations across tasks to improve performance on any single task. While sharing representations is an important mechanism to share information across tasks, its success depends on how well the structure underlying the tasks is captured. In some real-world situations, we have access to metadata, or additional information about a task, that may not provide any new insight in the context of a single task setup alone but inform relations across multiple tasks. While this metadata can be useful for improving multi-task learning performance, effectively incorporating it can be an additional challenge. We posit that an efficient approach to knowledge transfer is through the use of multiple context-dependent, composable representations shared across a family of tasks. In this framework, metadata can help to learn interpretable representations and provide the context to inform which representations to compose and how to compose them. We use the proposed approach to obtain state-of-the-art results in Meta-World, a challenging multi-task benchmark consisting of 50 distinct robotic manipulation tasks.

arXiv commentAccepted at the 38th International Conference on Machine Learning (ICML 2021). 17 pages, 4 figures, 20 tables

References in corpus (23)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• An Overview of Multi-Task Learning in Deep Neural Networks
• Model-Based Reinforcement Learning for Atari
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Multi-Object Representation Learning with Iterative Variational Inference
• Modular Multitask Reinforcement Learning with Policy Sketches
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Grounded Language Learning in a Simulated 3D World
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• Gradient Surgery for Multi-Task Learning
• The Benefit of Multitask Representation Learning
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Adapting Auxiliary Losses Using Gradient Similarity
• Learning Independent Causal Mechanisms
• Policy Distillation
• Recurrent Independent Mechanisms
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Automatically Composing Representation Transformations as a Means for Generalization
• Causal Discovery in Physical Systems from Videos
• Regularizing Deep Multi-Task Networks using Orthogonal Gradients
• Intervention Design for Effective Sim2Real Transfer

Cited by in corpus (7)

• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Block Contextual MDPs for Continual Learning
• Interleaved Multitask Learning with Energy Modulated Learning Progress
• Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making