Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning

Abstract

Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introduces the deep implicit coordination graph (DICG) architecture for such scenarios. DICG consists of a module for inferring the dynamic coordination graph structure which is then used by a graph neural network based module to learn to implicitly reason about the joint actions or values. DICG allows learning the tradeoff between full centralization and decentralization via standard actor-critic methods to significantly improve coordination for domains with large number of agents. We apply DICG to both centralized-training-centralized-execution and centralized-training-decentralized-execution regimes. We demonstrate that DICG solves the relative overgeneralization pathology in predatory-prey tasks as well as outperforms various MARL baselines on the challenging StarCraft II Multi-agent Challenge (SMAC) and traffic junction environments.

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Semi-Supervised Classification with Graph Convolutional Networks
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• StarCraft II: A New Challenge for Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Learning Attentional Communication for Multi-Agent Cooperation
• Long Short-Term Memory-Networks for Machine Reading
• Relational Deep Reinforcement Learning
• Multiagent Soft Q-Learning
• Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks
• Deep Coordination Graphs
• Deep Multi-Agent Reinforcement Learning with Relevance Graphs
• Multi-Agent Game Abstraction via Graph Attention Neural Network
• Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
• Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning

Cited by in corpus (5)

• Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing
• Evaluating Generalization and Transfer Capacity of Multi-Agent Reinforcement Learning Across Variable Number of Agents