Multi-Agent Common Knowledge Reinforcement Learning

Abstract

Cooperative multi-agent reinforcement learning often requires decentralised policies, which severely limit the agents' ability to coordinate their behaviour. In this paper, we show that common knowledge between agents allows for complex decentralised coordination. Common knowledge arises naturally in a large number of decentralised cooperative multi-agent tasks, for example, when agents can reconstruct parts of each others' observations. Since agents an independently agree on their common knowledge, they can execute complex coordinated policies that condition on this knowledge in a fully decentralised fashion. We propose multi-agent common knowledge reinforcement learning (MACKRL), a novel stochastic actor-critic algorithm that learns a hierarchical policy tree. Higher levels in the hierarchy coordinate groups of agents by conditioning on their common knowledge, or delegate to lower levels with smaller subgroups but potentially richer common knowledge. The entire policy tree can be executed in a fully decentralised fashion. As the lowest policy tree level consists of independent policies for each agent, MACKRL reduces to independently learnt decentralised policies as a special case. We demonstrate that our method can exploit common knowledge for superior performance on complex decentralised coordination tasks, including a stochastic matrix game and challenging problems in StarCraft II unit micromanagement.

arXiv commentAdvances in Neural Information Processing Systems, 9924-9935

References in corpus (18)

• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• StarCraft II: A New Challenge for Reinforcement Learning
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
• Cooperative and Distributed Reinforcement Learning of Drones for Field Coverage
• Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
• Learning to Coordinate Efficiently: A Model-based Approach
• Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning
• Learning to Communicate Implicitly By Actions
• Learning for Multi-robot Cooperation in Partially Observable Stochastic Environments with Macro-actions
• Multi-vehicle Flocking Control with Deep Deterministic Policy Gradient Method

Cited by in corpus (6)

• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• Navigating the Landscape of Multiplayer Games
• Open Problems in Cooperative AI
• Decentralized Multi-Agents by Imitation of a Centralized Controller
• Spatial Influence-aware Reinforcement Learning for Intelligent Transportation System