SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference

Abstract

We present a modern scalable reinforcement learning agent called SEED (Scalable, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions of frames per second but also to lower the cost of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state of the art distributed algorithms, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and is evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 three times faster in wall-time. For the scenarios we consider, a 40% to 80% cost reduction for running experiments is achieved. The implementation along with experiments is open-sourced so results can be reproduced and novel ideas tried out.

New version that includes changes made during the ICLR 2020 review process (https://openreview.net/forum?id=rkgvXlrKwH)

References in corpus (8)

• StarCraft II: A New Challenge for Reinforcement Learning
• Revisiting Distributed Synchronous SGD
• Large Batch Training of Convolutional Networks
• Massively Parallel Methods for Deep Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• An Empirical Model of Large-Batch Training
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• TorchBeast: A PyTorch Platform for Distributed RL

Cited by in corpus (3)

• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• VALAN: Vision and Language Agent Navigation