Distributed Prioritized Experience Replay

Abstract

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

arXiv commentAccepted to International Conference on Learning Representations 2018

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• One weird trick for parallelizing convolutional neural networks
• Emergence of Locomotion Behaviours in Rich Environments
• Massively Parallel Methods for Deep Reinforcement Learning
• Online Batch Selection for Faster Training of Neural Networks
• Variance Reduction in SGD by Distributed Importance Sampling
• Efficient Parallel Methods for Deep Reinforcement Learning

Cited by in corpus (136)

• Dota 2 with Large Scale Deep Reinforcement Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Distributed Distributional Deterministic Policy Gradients
• Challenges of Real-World Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Ray: A Distributed Framework for Emerging AI Applications
• Exploration by Random Network Distillation
• Agent57: Outperforming the Atari Human Benchmark
• Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Go-Explore: a New Approach for Hard-Exploration Problems
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Dopamine: A Research Framework for Deep Reinforcement Learning
• An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• A Survey of Deep Reinforcement Learning in Video Games
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• An Empirical Model of Large-Batch Training
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Scalable agent alignment via reward modeling: a research direction
• Reinforcement Learning for Ridesharing: An Extended Survey
• Unsupervised Learning of Object Keypoints for Perception and Control
• Deep Reinforcement Learning and the Deadly Triad
• Randomized Prior Functions for Deep Reinforcement Learning
• Accelerated Methods for Deep Reinforcement Learning
• CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms
• Observe and Look Further: Achieving Consistent Performance on Atari
• Never Give Up: Learning Directed Exploration Strategies
• Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A Survey
• Towards Characterizing Divergence in Deep Q-Learning
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• An empirical investigation of the challenges of real-world reinforcement learning
• Softmax Deep Double Deterministic Policy Gradients
• Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• Curriculum-based Reinforcement Learning for Distribution System Critical Load Restoration
• Behaviour Suite for Reinforcement Learning
• Prioritized Sequence Experience Replay
• Mastering Atari with Discrete World Models
• ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problems
• Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
• TF-Replicator: Distributed Machine Learning for Researchers
• Exploration-Enhanced POLITEX
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Experience Replay with Likelihood-free Importance Weights
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Accelerating Reinforcement Learning through GPU Atari Emulation
• Modern Deep Reinforcement Learning Algorithms
• Learn to Interpret Atari Agents
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Synthetic Returns for Long-Term Credit Assignment
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• An Open-Source Framework for Adaptive Traffic Signal Control
• Adversarial Imitation Learning via Random Search
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning
• Learning Awareness Models
• Positive-Unlabeled Reward Learning
• Dynamic Experience Replay
• Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Training Larger Networks for Deep Reinforcement Learning
• Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs
• Affordance-based Reinforcement Learning for Urban Driving
• A Deep Reinforcement Learning Approach for Global Routing
• Direct and indirect reinforcement learning
• Adapting Behaviour for Learning Progress
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• DMRO:A Deep Meta Reinforcement Learning-based Task Offloading Framework for Edge-Cloud Computing
• The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning
• Renewable Energy Powered and Open RAN-based Architecture for 5G Fixed Wireless Access Provisioning in Rural Areas
• Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization
• Digital Twin Backed Closed-Loops for Energy-Aware and Open RAN-based Fixed Wireless Access Serving Rural Areas
• QoS-Aware Scheduling in New Radio Using Deep Reinforcement Learning
• Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World
• Reverb: A Framework For Experience Replay
• Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
• Multiplayer Support for the Arcade Learning Environment
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning
• Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning
• Taylor Expansion Policy Optimization
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Levels of Analysis for Machine Learning
• Off-Policy Actor-Critic with Shared Experience Replay
• Sample-Efficient Automated Deep Reinforcement Learning
• Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Disentangling causal effects for hierarchical reinforcement learning
• Action Set Based Policy Optimization for Safe Power Grid Management
• Distributed Soft Actor-Critic with Multivariate Reward Representation and Knowledge Distillation
• Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning
• Competitive Experience Replay
• Agent Environment Cycle Games
• Distributed Deep Reinforcement Learning: An Overview
• Off-Belief Learning
• Enhanced detection of fetal pose in 3D MRI by Deep Reinforcement Learning with physical structure priors on anatomy
• Simulating multi-exit evacuation using deep reinforcement learning
• Group Equivariant Deep Reinforcement Learning
• Policy Optimization Reinforcement Learning with Entropy Regularization
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience
• IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
• Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving
• Convolutional Reservoir Computing for World Models
• Reinforcement Learning with Convolutional Reservoir Computing
• SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
• Regularly Updated Deterministic Policy Gradient Algorithm
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Distributed Multi-Agent Deep Reinforcement Learning Framework for Whole-building HVAC Control
• Improving Experience Replay through Modeling of Similar Transitions' Sets
• Evolutionary Self-Replication as a Mechanism for Producing Artificial Intelligence
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Reinforcement Learning Control of Robotic Knee with Human in the Loop by Flexible Policy Iteration
• Improvements on Hindsight Learning
• Constrained Policy Improvement for Safe and Efficient Reinforcement Learning
• Deep Reinforcement Learning for Field Development Optimization
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy
• Reinforcement Learning for Robust Missile Autopilot Design
• When should agents explore?
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning
• Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations
• An investigation of model-free planning
• Large Batch Experience Replay
• High Throughput Training of Deep Surrogates from Large Ensemble Runs
• Locality-Sensitive Experience Replay for Online Recommendation
• Proximal Policy Optimization with Mixed Distributed Training
• Human-Level Control without Server-Grade Hardware