Sharing Knowledge in Multi-Task Deep Reinforcement Learning

Abstract

We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.

References in corpus (6)

• Continuous control with deep reinforcement learning
• Theoretical Models of Learning to Learn
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• The Benefit of Multitask Representation Learning
• Sample Complexity of Multi-task Reinforcement Learning
• Transfer from Multiple MDPs

Cited by in corpus (23)

• Invariant Causal Prediction for Block MDPs
• Deep Reinforcement and InfoMax Learning
• Multi-Task Reinforcement Learning with Context-based Representations
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Multi-task Batch Reinforcement Learning with Metric Learning
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
• Near-optimal Representation Learning for Linear Bandits and Linear RL
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs
• On the Power of Multitask Representation Learning in Linear MDP
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• Impact of Representation Learning in Linear Bandits
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Randomized Value Functions via Posterior State-Abstraction Sampling
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning
• A Conceptual Framework for Lifelong Learning
• Provably Efficient Multi-Task Reinforcement Learning with Model Transfer
• Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization
• PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processes
• Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making
• Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning
• Multitasking Inhibits Semantic Drift