SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference

Abstract

We present a modern scalable reinforcement learning agent called SEED (Scalable, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions of frames per second but also to lower the cost of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state of the art distributed algorithms, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and is evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 three times faster in wall-time. For the scenarios we consider, a 40% to 80% cost reduction for running experiments is achieved. The implementation along with experiments is open-sourced so results can be reproduced and novel ideas tried out.

New version that includes changes made during the ICLR 2020 review process (https://openreview.net/forum?id=rkgvXlrKwH)

References in corpus (20)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Revisiting Distributed Synchronous SGD
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Large Batch Training of Convolutional Networks
• Massively Parallel Methods for Deep Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• An Empirical Model of Large-Batch Training
• Accelerated Methods for Deep Reinforcement Learning
• Observe and Look Further: Achieving Consistent Performance on Atari
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• TorchBeast: A PyTorch Platform for Distributed RL
• Fast Task Inference with Variational Intrinsic Successor Features
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Neural Replicator Dynamics
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Interval timing in deep reinforcement learning agents
• Cross-View Policy Learning for Street Navigation

Cited by in corpus (16)

• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• AI in Human-computer Gaming: Techniques, Challenges and Opportunities
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• VALAN: Vision and Language Agent Navigation
• Agent-Centric Representations for Multi-Agent Reinforcement Learning
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Distributed Deep Reinforcement Learning: An Overview
• WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Deep reinforcement learning for RAN optimization and control