Accelerated Methods for Deep Reinforcement Learning

Abstract

Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari games in mere minutes, using both synchronous and asynchronous algorithms.

arXiv commentv2: -Added game performance statistics summary for algorithm scaling across full Atari game set. -Added full set of learning curves (appendix). -Fixed images to remove phantom borders. -Streamlined some discussion, moved some details to appendix

References in corpus (8)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Dueling Network Architectures for Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• Distributed Prioritized Experience Replay
• Massively Parallel Methods for Deep Reinforcement Learning
• Efficient Parallel Methods for Deep Reinforcement Learning

Cited by in corpus (27)

• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• An Empirical Model of Large-Batch Training
• Tianshou: a Highly Modularized Deep Reinforcement Learning Library
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Accelerating Reinforcement Learning through GPU Atari Emulation
• Accelerated Deep Reinforcement Learning Based Load Shedding for Emergency Voltage Control
• Asynchronous Methods for Model-Based Reinforcement Learning
• Training Larger Networks for Deep Reinforcement Learning
• Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training
• Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
• Scaling Scaling Laws with Board Games
• Marathon Environments: Multi-Agent Continuous Control Benchmarks in a Modern Video Game Engine
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
• The Architectural Implications of Distributed Reinforcement Learning on CPU-GPU Systems
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments
• Robot_gym: accelerated robot training through simulation in the cloud with ROS and Gazebo
• Reusability and Transferability of Macro Actions for Reinforcement Learning
• Human-Level Control without Server-Grade Hardware
• Progressive extension of reinforcement learning action dimension for asymmetric assembly tasks