Multi-task Deep Reinforcement Learning with PopArt

Abstract

The reinforcement learning community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained one task at the time, each new task requiring to train a brand new agent instance. This means the learning algorithm is general, but each solution is not; each agent can only solve the one task it was trained on. In this work, we study the problem of learning to master not one but multiple sequential-decision tasks at once. A general issue in multi-task learning is that a balance must be found between the needs of multiple tasks competing for the limited resources of a single learning system. Many learning algorithms can get distracted by certain tasks in the set of tasks to solve. Such tasks appear more salient to the learning process, for instance because of the density or magnitude of the in-task rewards. This causes the algorithm to focus on those salient tasks at the expense of generality. We propose to automatically adapt the contribution of each task to the agent's updates, so that all tasks have a similar impact on the learning dynamics. This resulted in state of the art performance on learning to play all games in a set of 57 diverse Atari games. Excitingly, our method learned a single trained policy - with a single set of weights - that exceeds median human performance. To our knowledge, this was the first time a single agent surpassed human-level performance on this multi-task domain. The same approach also demonstrated state of the art performance on a set of 30 tasks in the 3D reinforcement learning platform DeepMind Lab.

References in corpus (17)

• Continuous control with deep reinforcement learning
• Overcoming catastrophic forgetting in neural networks
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Benchmarking Deep Reinforcement Learning for Continuous Control
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Deep Exploration via Bootstrapped DQN
• Massively Parallel Methods for Deep Reinforcement Learning
• Learning to Navigate in Complex Environments
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Distral: Robust Multitask Reinforcement Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Learning values across many orders of magnitude
• Sample Complexity of Multi-task Reinforcement Learning
• A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
• Kickstarting Deep Reinforcement Learning
• DeepMind Lab
• Unicorn: Continual Learning with a Universal, Off-policy Agent

Cited by in corpus (20)

• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Stabilizing Transformers for Reinforcement Learning
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Adapting Auxiliary Losses Using Gradient Similarity
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Meta-Learning surrogate models for sequential decision making
• Disentangled Skill Embeddings for Reinforcement Learning
• Zero-Shot Terrain Generalization for Visual Locomotion Policies
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• State of the Art in Fair ML: From Moral Philosophy and Legislation to Fair Classifiers
• Multi-Agent Reinforcement Learning for Unmanned Aerial Vehicle Coordination by Multi-Critic Policy Gradient Optimization
• Importance Weighted Evolution Strategies
• Multi-task Reinforcement Learning with a Planning Quasi-Metric
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• RLCache: Automated Cache Management Using Reinforcement Learning
• Utilizing Skipped Frames in Action Repeats via Pseudo-Actions