Deep Coordination Graphs
arXiv:1910.00091
Abstract
This paper introduces the deep coordination graph (DCG) for collaborative multi-agent reinforcement learning. DCG strikes a flexible trade-off between representational capacity and generalization by factoring the joint value function of all agents according to a coordination graph into payoffs between pairs of agents. The value can be maximized by local message passing along the graph, which allows training of the value function end-to-end with Q-learning. Payoff functions are approximated with deep neural networks that employ parameter sharing and low-rank approximations to significantly improve sample efficiency. We show that DCG can solve predator-prey tasks that highlight the relative overgeneralization pathology, as well as challenging StarCraft II micromanagement tasks.
arXiv commentAccepted at ICML 2020
References in corpus (11)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Loopy Belief Propagation for Approximate Inference: An Empirical Study
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Counterfactual Multi-Agent Policy Gradients
- Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Mean Field Multi-Agent Reinforcement Learning
- Loopy Belief Propagation as a Basis for Communication in Sensor Networks
- Multiagent Soft Q-Learning
- Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
Cited by in corpus (9)
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
- FACMAC: Factored Multi-Agent Centralised Policy Gradients
- Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
- Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
- Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
- Scalable Anytime Planning for Multi-Agent MDPs
- UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
- Mimicking Evolution with Reinforcement Learning