RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
arXiv:2011.12719
Abstract
Researchers and practitioners in the field of reinforcement learning (RL) frequently leverage parallel computation, which has led to a plethora of new algorithms and systems in the last few years. In this paper, we re-examine the challenges posed by distributed RL and try to view it through the lens of an old idea: distributed dataflow. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL, and validate its practicality by porting the full suite of algorithms in RLlib, a widely adopted distributed RL library. Concretely, RLlib Flow provides 2-9 code savings in real production code and enables the composition of multi-agent algorithms not possible by end users before. The open-source code is available as part of RLlib at https://github.com/ray-project/ray/tree/master/rllib.
arXiv commentNeurIPS 2021. The first two authors contributed equally to this work
References in corpus (11)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Soft Actor-Critic Algorithms and Applications
- Benchmarking Deep Reinforcement Learning for Continuous Control
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Distributed Prioritized Experience Replay
- Massively Parallel Methods for Deep Reinforcement Learning
- Dopamine: A Research Framework for Deep Reinforcement Learning
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Accelerated Methods for Deep Reinforcement Learning
- Acme: A Research Framework for Distributed Reinforcement Learning
- SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning