Accelerating Reinforcement Learning through GPU Atari Emulation

Abstract

We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep reinforcement algorithms. CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU parallelization to run thousands of games simultaneously and it renders frames directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU communication bandwidth. CuLE generates up to 155M frames per hour on a single GPU, a finding previously achieved only through a cluster of CPUs. Beyond highlighting the differences between CPU and GPU emulators in the context of reinforcement learning, we show how to leverage the high throughput of CuLE by effective batching of the training data, and show accelerated convergence for A2C+V-trace. CuLE is available at https://github.com/NVLabs/cule .

References in corpus (10)

• Continuous control with deep reinforcement learning
• Prioritized Experience Replay
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• DeepMind Control Suite
• Distributed Prioritized Experience Replay
• Massively Parallel Methods for Deep Reinforcement Learning
• Learning to Navigate in Complex Environments
• Accelerated Methods for Deep Reinforcement Learning
• Efficient Parallel Methods for Deep Reinforcement Learning

Cited by in corpus (6)

• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Scaling Scaling Laws with Board Games
• Stochastic Optimization with Laggard Data Pipelines
• WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU
• Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction
• OpenGraphGym-MG: Using Reinforcement Learning to Solve Large Graph Optimization Problems on MultiGPU Systems