Learning Nearly Decomposable Value Functions Via Communication Minimization

Abstract

Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than $80\%$ of communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndq.

arXiv comment8th International Conference on Learning Representations (ICLR 2020)

References in corpus (12)

• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Multi-Agent Cooperation and the Emergence of (Natural) Language
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

Cited by in corpus (7)

• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Robust Multi-agent Communication via Multi-view Message Certification
• Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• SOAC: The Soft Option Actor-Critic Architecture
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates