Massively Parallel Methods for Deep Reinforcement Learning

Abstract

We present the first massively distributed architecture for deep reinforcement learning. This architecture uses four main components: parallel actors that generate new behaviour; parallel learners that are trained from stored experience; a distributed neural network to represent the value function or behaviour policy; and a distributed store of experience. We used our architecture to implement the Deep Q-Network algorithm (DQN). Our distributed algorithm was applied to 49 games from Atari 2600 games from the Arcade Learning Environment, using identical hyperparameters. Our performance surpassed non-distributed DQN in 41 of the 49 games and also reduced the wall-time required to achieve these results by an order of magnitude on most games.

Presented at the Deep Learning Workshop, International Conference on Machine Learning, Lille, France, 2015

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Playing Atari with Deep Reinforcement Learning
• Going Deeper with Convolutions

Cited by in corpus (115)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• A Brief Survey of Deep Reinforcement Learning
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Emergence of Locomotion Behaviours in Rich Environments
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Distributed Prioritized Experience Replay
• Deep Exploration via Bootstrapped DQN
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Ray: A Distributed Framework for Emerging AI Applications
• Implicit Quantile Networks for Distributional Reinforcement Learning
• A Distributional Perspective on Reinforcement Learning
• A Study on Overfitting in Deep Reinforcement Learning
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Go-Explore: a New Approach for Hard-Exploration Problems
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Deep Attention Recurrent Q-Network
• Accelerated Methods for Deep Reinforcement Learning
• Multi-Objective Deep Reinforcement Learning
• Deep Successor Reinforcement Learning
• Efficient Parallel Methods for Deep Reinforcement Learning
• Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks
• How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
• Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
• Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
• Acme: A Research Framework for Distributed Reinforcement Learning
• Open-Sourced Reinforcement Learning Environments for Surgical Robotics
• Combining Deep Reinforcement Learning and Safety Based Control for Autonomous Driving
• Distributed Deep Q-Learning
• Graying the black box: Understanding DQNs
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
• Learning how to Active Learn: A Deep Reinforcement Learning Approach
• AI in Human-computer Gaming: Techniques, Challenges and Opportunities
• Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup
• Behaviour Suite for Reinforcement Learning
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learning
• Uncovering Instabilities in Variational-Quantum Deep Q-Networks
• Multi-focus Attention Network for Efficient Deep Reinforcement Learning
• Online Machine Learning in Big Data Streams
• Multi-task Deep Reinforcement Learning with PopArt
• Safe Option-Critic: Learning Safety in the Option-Critic Architecture
• Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
• A Data-Efficient Deep Learning Approach for Deployable Multimodal Social Robots
• Analysing Results from AI Benchmarks: Key Indicators and How to Obtain Them
• Locally Private Distributed Reinforcement Learning
• Collaborative Deep Reinforcement Learning
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Accelerating Reinforcement Learning through GPU Atari Emulation
• Tracking-by-Trackers with a Distilled and Reinforced Model
• Real-Time Machine Learning: The Missing Pieces
• Asynchronous Methods for Model-Based Reinforcement Learning
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Learn to Interpret Atari Agents
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Deep Reinforcement Learning With Macro-Actions
• Podracer architectures for scalable Reinforcement Learning
• Learning Runtime Parameters in Computer Systems with Delayed Experience Injection
• The Benchmark Lottery
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
• Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms
• Population-Guided Parallel Policy Search for Reinforcement Learning
• Using Unity to Help Solve Intelligence
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• Idiosyncrasies and challenges of data driven learning in electronic trading
• A Survey of Exploration Methods in Reinforcement Learning
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• MAIDCRL: Semi-centralized Multi-Agent Influence Dense-CNN Reinforcement Learning
• StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
• Grounding Language for Transfer in Deep Reinforcement Learning
• Taylor Expansion Policy Optimization
• Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation
• Off-Policy Actor-Critic with Shared Experience Replay
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
• Relationship Explainable Multi-objective Reinforcement Learning with Semantic Explainability Generation
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning
• QEDCartographer: Automating Formal Verification Using Reward-Free Reinforcement Learning
• Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning
• On Catastrophic Interference in Atari 2600 Games
• Spark-MPI: Approaching the Fifth Paradigm of Cognitive Applications
• Online Meta-learning by Parallel Algorithm Competition
• Group-Agent Reinforcement Learning
• Deep In-GPU Experience Replay
• Distributed Deep Reinforcement Learning: An Overview
• Deep Reinforcement Learning for Playing 2.5D Fighting Games
• Robo-PlaNet: Learning to Poke in a Day
• Amanuensis: The Programmer's Apprentice
• IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
• Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT
• Coding for Distributed Multi-Agent Reinforcement Learning
• Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning
• Improved Learning in Evolution Strategies via Sparser Inter-Agent Network Topologies
• Discrete-to-Deep Supervised Policy Learning
• APPLR: Adaptive Planner Parameter Learning from Reinforcement
• SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
• Deep-Reinforcement-Learning-Based Distributed Vehicle Position Controls for Coverage Expansion in mmWave V2X
• Parallelized Reverse Curriculum Generation
• Human-Level Control without Server-Grade Hardware
• High Throughput Training of Deep Surrogates from Large Ensemble Runs
• Communication Efficient Parallel Reinforcement Learning
• Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning
• A Self-Taught Artificial Agent for Multi-Physics Computational Model Personalization
• Biological Blueprints for Next Generation AI Systems
• How to Organize your Deep Reinforcement Learning Agents: The Importance of Communication Topology