SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning

Abstract

We present an overview of SURREAL-System, a reproducible, flexible, and scalable framework for distributed reinforcement learning (RL). The framework consists of a stack of four layers: Provisioner, Orchestrator, Protocol, and Algorithms. The Provisioner abstracts away the machine hardware and node pools across different cloud providers. The Orchestrator provides a unified interface for scheduling and deploying distributed algorithms by high-level description, which is capable of deploying to a wide range of hardware from a personal laptop to full-fledged cloud clusters. The Protocol provides network communication primitives optimized for RL. Finally, the SURREAL algorithms, such as Proximal Policy Optimization (PPO) and Evolution Strategies (ES), can easily scale to 1000s of CPU cores and 100s of GPUs. The learning performances of our distributed algorithms establish new state-of-the-art on OpenAI Gym and Robotics Suites tasks.

Technical report of the SURREAL system. See more details at https://surreal.stanford.edu

References in corpus (10)

• Trust Region Policy Optimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Distributed Prioritized Experience Replay
• Massively Parallel Methods for Deep Reinforcement Learning
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow