IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Abstract

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach.

Cited by in corpus (270)

• Addressing Function Approximation Error in Actor-Critic Methods
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Dota 2 with Large Scale Deep Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• A Survey on Multi-Task Learning
• Unity: A General Platform for Intelligent Agents
• Multi-Task Learning with Deep Neural Networks: A Survey
• Experience Replay for Continual Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Challenges of Real-World Reinforcement Learning
• Dynamic Scheduling for Stochastic Edge-Cloud Computing Environments using A3C learning and Residual Recurrent Neural Networks
• Exploration by Random Network Distillation
• The Hanabi Challenge: A New Frontier for AI Research
• Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach
• First return, then explore
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Go-Explore: a New Approach for Hard-Exploration Problems
• Quantifying Generalization in Reinforcement Learning
• Episodic Curiosity through Reachability
• Ablation Studies in Artificial Neural Networks
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Relational Deep Reinforcement Learning
• Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Scalable agent alignment via reward modeling: a research direction
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Reinforcement Learning Algorithms: An Overview and Classification
• Accelerated Methods for Deep Reinforcement Learning
• Generalization and Regularization in DQN
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Adapting Auxiliary Losses Using Gradient Similarity
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• Learning Montezuma's Revenge from a Single Demonstration
• Never Give Up: Learning Directed Exploration Strategies
• Reinforcement Learning for Integer Programming: Learning to Cut
• Hierarchical visuomotor control of humanoids
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Discovering Reinforcement Learning Algorithms
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Environmental drivers of systematicity and generalization in a situated agent
• Learning to Navigate in Cities Without a Map
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Phasic Policy Gradient
• The StreetLearn Environment and Dataset
• Dealing with Sparse Rewards in Reinforcement Learning
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• The NetHack Learning Environment
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Imitating Interactive Intelligence
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Domain Generalization with MixStyle
• Kickstarting Deep Reinforcement Learning
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
• Distilling Policy Distillation
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• AI in Human-computer Gaming: Techniques, Challenges and Opportunities
• Grounded Language Learning Fast and Slow
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• MVFST-RL: An Asynchronous RL Framework for Congestion Control with Delayed Actions
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• A Self-Tuning Actor-Critic Algorithm
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Q-Learning in enormous action spaces via amortized approximate maximization
• Mix&Match - Agent Curricula for Reinforcement Learning
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• TorchBeast: A PyTorch Platform for Distributed RL
• At Human Speed: Deep Reinforcement Learning with Action Delay
• Relational recurrent neural networks
• Towards Interpretable Reinforcement Learning Using Attention Augmented Agents
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Mastering Complex Control in MOBA Games with Deep Reinforcement Learning
• Multi-task Deep Reinforcement Learning with PopArt
• Meta-trained agents implement Bayes-optimal agents
• Universal Successor Features Approximators
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• General Value Function Networks
• TF-Replicator: Distributed Machine Learning for Researchers
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Accelerating Quadratic Optimization with Reinforcement Learning
• Fast Task Inference with Variational Intrinsic Successor Features
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Experience Replay with Likelihood-free Importance Weights
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• Regularization Matters in Policy Optimization
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Dynamics-aware Embeddings
• From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization
• Causally Correct Partial Models for Reinforcement Learning
• Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
• Accelerating Reinforcement Learning through GPU Atari Emulation
• Shaping Belief States with Generative Environment Models for RL
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• RTFM: Generalising to Novel Environment Dynamics via Reading
• Optimization Methods for Interpretable Differentiable Decision Trees in Reinforcement Learning
• Auxiliary Tasks and Exploration Enable ObjectNav
• Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Online Off-policy Prediction
• Tracking-by-Trackers with a Distilled and Reinforced Model
• Remember and Forget for Experience Replay
• Asynchronous Methods for Model-Based Reinforcement Learning
• FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Synthetic Returns for Long-Term Credit Assignment
• Muesli: Combining Improvements in Policy Optimization
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• An Open-Source Framework for Adaptive Traffic Signal Control
• Feature Partitioning for Efficient Multi-Task Architectures
• Neural Replicator Dynamics
• WordCraft: An Environment for Benchmarking Commonsense Agents
• Meta-Learning surrogate models for sequential decision making
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences
• Divergence-Augmented Policy Optimization
• Human-Level Performance in No-Press Diplomacy via Equilibrium Search
• Generalizing from a few environments in safety-critical reinforcement learning
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Direct and indirect reinforcement learning
• Adapting Behaviour for Learning Progress
• MVP: Unified Motion and Visual Self-Supervised Learning for Large-Scale Robotic Navigation
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• An Empirical Study on Google Research Football Multi-agent Scenarios
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Reinforcement Learning Generalization with Surprise Minimization
• Geometric Entropic Exploration
• Evaluating Soccer Player: from Live Camera to Deep Reinforcement Learning
• AndroidEnv: A Reinforcement Learning Platform for Android
• Agent-Centric Representations for Multi-Agent Reinforcement Learning
• Long-Term Planning and Situational Awareness in OpenAI Five
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
• Scaling Imitation Learning in Minecraft
• Solutions to problems with deep learning
• Local Search for Policy Iteration in Continuous Control
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
• A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
• Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization
• Finding online neural update rules by learning to remember
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Reverb: A Framework For Experience Replay
• Idiosyncrasies and challenges of data driven learning in electronic trading
• Causal Analysis of Agent Behavior for AI Safety
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environments
• Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Using Unity to Help Solve Intelligence
• Generalized Off-Policy Actor-Critic
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Neural Packet Classification
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Probing Emergent Semantics in Predictive Agents via Question Answering
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• Empirical Likelihood for Contextual Bandits
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• Discovery of Options via Meta-Learned Subgoals
• Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals
• Taylor Expansion Policy Optimization
• Playing Minecraft with Behavioural Cloning
• Sample-Efficient Automated Deep Reinforcement Learning
• Self-play Learning Strategies for Resource Assignment in Open-RAN Networks
• Importance Resampling for Off-policy Prediction
• Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
• Off-Policy Actor-Critic with Shared Experience Replay
• Compressed Federated Reinforcement Learning with a Generative Model
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
• Evaluating the Rainbow DQN Agent in Hanabi with Unseen Partners
• Acting in Delayed Environments with Non-Stationary Markov Policies
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Learning Representations in Reinforcement Learning:An Information Bottleneck Approach
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• Towards Automatic Actor-Critic Solutions to Continuous Control
• On Catastrophic Interference in Atari 2600 Games
• Adaptive Transformers in RL
• Contextual Recurrent Neural Networks
• Reinforcement Learning Trajectory Generation and Control for Aggressive Perching on Vertical Walls with Quadrotors
• Distributed Deep Reinforcement Learning: An Overview
• TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• SegVisRL: Visuomotor Development for a Lunar Rover for Hazard Avoidance using Camera Images
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Importance Weighted Evolution Strategies
• Direct Advantage Estimation
• A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
• Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learning
• Policy Optimization Reinforcement Learning with Entropy Regularization
• RLgraph: Modular Computation Graphs for Deep Reinforcement Learning
• On Proximal Policy Optimization's Heavy-tailed Gradients
• System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Reinforcement Learning with Latent Flow
• Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
• Towards Modular Algorithm Induction
• Learning Policies through Quantile Regression
• Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning
• Towards Combining On-Off-Policy Methods for Real-World Applications
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Instance based Generalization in Reinforcement Learning
• End-to-End Egospheric Spatial Memory
• PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network
• WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning
• An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
• IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling
• Scaling shared model governance via model splitting
• Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations
• An Empirical Comparison of Off-policy Prediction Learning Algorithms in the Four Rooms Environment
• Applying supervised and reinforcement learning methods to create neural-network-based agents for playing StarCraft II
• No-Press Diplomacy from Scratch
• Negotiating Team Formation Using Deep Reinforcement Learning
• Self-Consistent Models and Values
• Episodic Memory for Learning Subjective-Timescale Models
• Model-free Policy Learning with Reward Gradients
• Understanding the Pathologies of Approximate Policy Evaluation when Combined with Greedification in Reinforcement Learning
• Hybrid Supervised Reinforced Model for Dialogue Systems
• Deep reinforcement learning for RAN optimization and control
• Sim-to-Real Transfer of Robot Learning with Variable Length Inputs
• Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Relational Mimic for Visual Adversarial Imitation Learning
• Learning to Superoptimize Real-world Programs
• Growing Action Spaces
• Adaptive Experience Selection for Policy Gradient
• Towards an Understanding of Default Policies in Multitask Policy Optimization
• Lyapunov-based uncertainty-aware safe reinforcement learning
• Logic and the $2$-Simplicial Transformer
• Rapid Task-Solving in Novel Environments
• An investigation of model-free planning
• Proximal Policy Optimization with Mixed Distributed Training
• A Study of Policy Gradient on a Class of Exactly Solvable Models
• Bootstrapped Meta-Learning
• An Exploration of Deep Learning Methods in Hungry Geese
• Deep Reinforcement Learning for Field Development Optimization