Learning to Schedule Communication in Multi-agent Reinforcement Learning

Abstract

Many real-world reinforcement learning tasks require multiple agents to make sequential decisions under the agents' interaction, where well-coordinated actions among the agents are crucial to achieve the target goal better at these tasks. One way to accelerate the coordination effect is to enable multiple agents to communicate with each other in a distributed manner and behave as a group. In this paper, we study a practical scenario when (i) the communication bandwidth is limited and (ii) the agents share the communication medium so that only a restricted number of agents are able to simultaneously use the medium, as in the state-of-the-art wireless networking standards. This calls for a certain form of communication scheduling. In that regard, we propose a multi-agent deep reinforcement learning framework, called SchedNet, in which agents learn how to schedule themselves, how to encode the messages, and how to select actions based on received messages. SchedNet is capable of deciding which agents should be entitled to broadcasting their (encoded) messages, by learning the importance of each agent's partially observed information. We evaluate SchedNet against multiple baselines under two different applications, namely, cooperative communication and navigation, and predator-prey. Our experiments show a non-negligible performance gap between SchedNet and other mechanisms such as the ones without communication and with vanilla scheduling methods, e.g., round robin, ranging from 32% to 43%.

Accepted in ICLR 2019

References in corpus (6)

• Continuous control with deep reinforcement learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Learning Attentional Communication for Multi-Agent Cooperation
• Learning Multiagent Communication with Backpropagation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols

Cited by in corpus (14)

• Shapley Q-value: A Local Reward Approach to Solve Global Reward Games
• Improving Coordination in Small-Scale Multi-Agent Deep Reinforcement Learning through Memory-driven Communication
• Learning Efficient Multi-agent Communication: An Information Bottleneck Approach
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• Collaborative Visual Navigation
• Model-based Reinforcement Learning for Service Mesh Fault Resiliency in a Web Application-level
• Learning Multi-agent Communication under Limited-bandwidth Restriction for Internet Packet Routing
• Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance
• Event-Triggered Multi-agent Reinforcement Learning with Communication under Limited-bandwidth Constraint
• Heterogeneous Graph Attention Networks for Learning Diverse Communication
• Multi-agent Cooperative Games Using Belief Map Assisted Training
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep Reinforcement Learning
• FireCommander: An Interactive, Probabilistic Multi-agent Environment for Heterogeneous Robot Teams