Acme: A Research Framework for Distributed Reinforcement Learning

Abstract

Deep reinforcement learning (RL) has led to many recent and groundbreaking advances. However, these advances have often come at the cost of both increased scale in the underlying architectures being trained as well as increased complexity of the RL algorithms used to train them. These increases have in turn made it more difficult for researchers to rapidly prototype new ideas or reproduce published RL algorithms. To address these concerns this work describes Acme, a framework for constructing novel RL algorithms that is specifically designed to enable agents that are built using simple, modular components that can be used at various scales of execution. While the primary goal of Acme is to provide a framework for algorithm development, a secondary goal is to provide simple reference implementations of important or state-of-the-art algorithms. These implementations serve both as a validation of our design decisions as well as an important contribution to reproducibility in RL research. In this work we describe the major design decisions made within Acme and give further details as to how its components can be used to implement various algorithms. Our experiments provide baselines for a number of common and state-of-the-art algorithms as well as showing how these algorithms can be scaled up for much larger and more complex environments. This highlights one of the primary advantages of Acme, namely that it can be used to implement large, distributed RL algorithms that can run at massive scales while still maintaining the inherent readability of that implementation. This work presents a second version of the paper which coincides with an increase in modularity, additional emphasis on offline, imitation and learning from demonstrations algorithms, as well as various new agents implemented as part of Acme.

arXiv commentThis work presents a second version of the paper which coincides with an increase in modularity, additional emphasis on offline, imitation and learning from demonstrations algorithms, as well as various new agents implemented as part of Acme

References in corpus (1)

• Massively Parallel Methods for Deep Reinforcement Learning

Cited by in corpus (36)

• Critic Regularized Regression
• DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM
• An empirical investigation of the challenges of real-world reinforcement learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Primal Wasserstein Imitation Learning
• AI in Human-computer Gaming: Techniques, Challenges and Opportunities
• Benchmarks for Deep Off-Policy Evaluation
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• What Matters for Adversarial Imitation Learning?
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Learning Deep Features in Instrumental Variable Regression
• On the role of planning in model-based deep reinforcement learning
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Residual Reinforcement Learning from Demonstrations
• AndroidEnv: A Reinforcement Learning Platform for Android
• Autonomous Blimp Control using Deep Reinforcement Learning
• Semi-supervised reward learning for offline reinforcement learning
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• Reverb: A Framework For Experience Replay
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Regularized Behavior Value Estimation
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
• Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• The Architectural Implications of Distributed Reinforcement Learning on CPU-GPU Systems
• Continuous Control with Action Quantization from Demonstrations
• WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU
• Evaluating model-based planning and planner amortization for continuous control
• SaLinA: Sequential Learning of Agents
• Implicitly Regularized RL with Implicit Q-Values
• Active Offline Policy Selection
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• OpenGraphGym-MG: Using Reinforcement Learning to Solve Large Graph Optimization Problems on MultiGPU Systems
• Efficient Reinforcement Learning Development with RLzoo
• Learning to Reach, Swim, Walk and Fly in One Trial: Data-Driven Control with Scarce Data and Side Information