Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation
arXiv:2103.04972
Abstract
Reinforcement learning in cooperative multi-agent settings has recently advanced significantly in its scope, with applications in cooperative estimation for advertising, dynamic treatment regimes, distributed control, and federated learning. In this paper, we discuss the problem of cooperative multi-agent RL with function approximation, where a group of agents communicates with each other to jointly solve an episodic MDP. We demonstrate that via careful message-passing and cooperative value iteration, it is possible to achieve near-optimal no-regret learning even with a fixed constant communication budget. Next, we demonstrate that even in heterogeneous cooperative settings, it is possible to achieve Pareto-optimal no-regret learning with limited communication. Our work generalizes several ideas from the multi-agent contextual and multi-armed bandit literature to MDPs and reinforcement learning.
arXiv comment53 pages including Appendix
References in corpus (13)
- Distributed Prioritized Experience Replay
- Federated Optimization in Heterogeneous Networks
- Distributed Learning in Multi-Armed Bandit with Multiple Players
- Massively Parallel Methods for Deep Reinforcement Learning
- Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- -Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
- Multi-Objective Deep Reinforcement Learning
- Efficient Parallel Methods for Deep Reinforcement Learning
- Distributed Exploration in Multi-Armed Bandits
- Learning Probability Measures with respect to Optimal Transport Metrics
- On Multi-Agent Learning in Team Sports Games
- No-regret Algorithms for Multi-task Bayesian Optimization