Distributed Deep Reinforcement Learning: An Overview

Abstract

Deep reinforcement learning (DRL) is a very active research area. However, several technical and scientific issues require to be addressed, amongst which we can mention data inefficiency, exploration-exploitation trade-off, and multi-task learning. Therefore, distributed modifications of DRL were introduced; agents that could be run on many machines simultaneously. In this article, we provide a survey of the role of the distributed approaches in DRL. We overview the state of the field, by studying the key research works that have a significant impact on how we can use distributed methods in DRL. We choose to overview these papers, from the perspective of distributed learning, and not the aspect of innovations in reinforcement learning algorithms. Also, we evaluate these methods on different tasks and compare their performance with each other and with single actor and learner agents.

arXiv comment15 pages, 9 figures

References in corpus (15)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Trust Region Policy Optimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Dueling Network Architectures for Deep Reinforcement Learning
• One weird trick for parallelizing convolutional neural networks
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• StarCraft II: A New Challenge for Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Massively Parallel Methods for Deep Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Emergent Complexity via Multi-Agent Competition
• Efficient Parallel Methods for Deep Reinforcement Learning
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference