The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning

Abstract

Many real-world problems require the coordination of multiple autonomous agents. Recent work has shown the promise of Graph Neural Networks (GNNs) to learn explicit communication strategies that enable complex multi-agent coordination. These works use models of cooperative multi-agent systems whereby agents strive to achieve a shared global goal. When considering agents with self-interested local objectives, the standard design choice is to model these as separate learning systems (albeit sharing the same environment). Such a design choice, however, precludes the existence of a single, differentiable communication channel, and consequently prohibits the learning of inter-agent communication strategies. In this work, we address this gap by presenting a learning model that accommodates individual non-shared rewards and a differentiable communication channel that is common among all agents. We focus on the case where agents have self-interested objectives, and develop a learning algorithm that elicits the emergence of adversarial communications. We perform experiments on multi-agent coverage and path planning problems, and employ a post-hoc interpretability technique to visualize the messages that agents communicate to each other. We show how a single self-interested agent is capable of learning highly manipulative communication strategies that allows it to significantly outperform a cooperative team of agents.

arXiv commentAccepted to Conference on Robot Learning (CoRL) 2020. Camera-ready version incorporating rebuttal

References in corpus (14)

• Neural Message Passing for Quantum Chemistry
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Interaction Networks for Learning about Objects, Relations and Physics
• Counterfactual Multi-Agent Policy Gradients
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Convolutional Neural Network Architectures for Signals Supported on Graphs
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• A multi-agent reinforcement learning model of common-pool resource appropriation
• Finding Friend and Foe in Multi-Agent Games
• No Press Diplomacy: Modeling Multi-Agent Gameplay
• Graph Neural Networks for Learning Robot Team Coordination

Cited by in corpus (6)

• Beyond Robustness: A Taxonomy of Approaches towards Resilient Multi-Robot Systems
• Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning
• Emergent Language: A Survey and Taxonomy
• Graph Neural Networks for Decentralized Multi-Robot Submodular Action Selection
• Multi-Agent Reinforcement Learning for Unmanned Aerial Vehicle Coordination by Multi-Critic Policy Gradient Optimization
• Multi-Agent Deep Reinforcement Learning For Persistent Monitoring With Sensing, Communication, and Localization Constraints