rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch

Abstract

Since the recent advent of deep reinforcement learning for game play and simulated robotic control, a multitude of new algorithms have flourished. Most are model-free algorithms which can be categorized into three families: deep Q-learning, policy gradients, and Q-value policy gradients. These have developed along separate lines of research, such that few, if any, code bases incorporate all three kinds. Yet these algorithms share a great depth of common deep reinforcement learning machinery. We are pleased to share rlpyt, which implements all three algorithm families on top of a shared, optimized infrastructure, in a single repository. It contains modular implementations of many common deep RL algorithms in Python using PyTorch, a leading deep learning library. rlpyt is designed as a high-throughput code base for small- to medium-scale research in deep RL. This white paper summarizes its features, algorithms implemented, and relation to prior work, and concludes with detailed implementation and usage notes. rlpyt is available at https://github.com/astooke/rlpyt.

v2: Updated learning curves for SAC and TD3, improved by bootstrapping value-function when trajectory ends due to time limit, and switching to newer SAC version, now referenced

References in corpus (18)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Soft Actor-Critic Algorithms and Applications
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Theano: new features and speed improvements
• cuDNN: Efficient Primitives for Deep Learning
• Distributed Prioritized Experience Replay
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Continuous Deep Q-Learning with Model-based Acceleration
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• An Empirical Model of Large-Batch Training
• Accelerated Methods for Deep Reinforcement Learning
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform

Cited by in corpus (31)

• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Tianshou: a Highly Modularized Deep Reinforcement Learning Library
• ChainerRL: A Deep Reinforcement Learning Library
• t-Soft Update of Target Network for Deep Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Quantum Reinforcement Learning: the Maze problem
• Pretraining Representations for Data-Efficient Reinforcement Learning
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Learning Shape Control of Elastoplastic Deformable Linear Objects
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning
• Learning to Manipulate Deformable Objects without Demonstrations
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Spatial Graph Attention and Curiosity-driven Policy for Antiviral Drug Discovery
• TAdam: A Robust Stochastic Gradient Optimizer
• The MAGICAL Benchmark for Robust Imitation
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning
• A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• Planning from Pixels using Inverse Dynamics Models
• Large Batch Simulation for Deep Reinforcement Learning
• Average-Reward Reinforcement Learning with Trust Region Methods
• Regularized Inverse Reinforcement Learning
• On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
• The AI Arena: A Framework for Distributed Multi-Agent Reinforcement Learning
• Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates
• Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations