TorchBeast: A PyTorch Platform for Distributed RL

Abstract

TorchBeast is a platform for reinforcement learning (RL) research in PyTorch. It implements a version of the popular IMPALA algorithm for fast, asynchronous, parallel training of RL agents. Additionally, TorchBeast has simplicity as an explicit design goal: We provide both a pure-Python implementation ("MonoBeast") as well as a multi-machine high-performance version ("PolyBeast"). In the latter, parts of the implementation are written in C++, but all parts pertaining to machine learning are kept in simple Python using PyTorch, with the environments provided using the OpenAI Gym interface. This enables researchers to conduct scalable RL research using TorchBeast without any programming knowledge beyond Python and PyTorch. In this paper, we describe the TorchBeast design principles and implementation and demonstrate that it performs on-par with IMPALA on Atari. TorchBeast is released as an open-source package under the Apache 2.0 license and is available at \url{https://github.com/facebookresearch/torchbeast}.

References in corpus (6)

• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• MVFST-RL: An Asynchronous RL Framework for Congestion Control with Delayed Actions
• High-Level Strategy Selection under Partial Observability in StarCraft: Brood War

Cited by in corpus (18)

• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• The NetHack Learning Environment
• MVFST-RL: An Asynchronous RL Framework for Congestion Control with Delayed Actions
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
• RTFM: Generalising to Novel Environment Dynamics via Reading
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• WordCraft: An Environment for Benchmarking Commonsense Agents
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• Large Batch Simulation for Deep Reinforcement Learning
• Guiding Global Placement With Reinforcement Learning
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Off-Belief Learning
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals
• Feudal Reinforcement Learning by Reading Manuals