A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
arXiv:1903.06372
Abstract
This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-agent version of emphatic temporal difference learning for off-policy policy evaluation, and proves convergence under linear function approximation. The paper then leverages this result, in conjunction with a novel multi-agent off-policy policy gradient theorem and recent work in both multi-agent on-policy and single-agent off-policy actor-critic methods, to develop and give convergence guarantees for a new multi-agent off-policy actor-critic algorithm.
References in corpus (9)
- Continuous control with deep reinforcement learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
- -Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
- Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
- Off-Policy Actor-Critic
- On Convergence of Emphatic Temporal-Difference Learning
- Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
- Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation