Multi-Agent Determinantal Q-Learning

Abstract

Centralized training with decentralized execution has become an important paradigm in multi-agent learning. Though practical, current methods rely on restrictive assumptions to decompose the centralized value function across agents for execution. In this paper, we eliminate this restriction by proposing multi-agent determinantal Q-learning. Our method is established on Q-DPP, an extension of determinantal point process (DPP) with partition-matroid constraint to multi-agent setting. Q-DPP promotes agents to acquire diverse behavioral models; this allows a natural factorization of the joint Q-functions with no need for \emph{a priori} structural constraints on the value function or special network architectures. We demonstrate that Q-DPP generalizes major solutions including VDN, QMIX, and QTRAN on decentralizable cooperative tasks. To efficiently draw samples from Q-DPP, we adopt an existing sample-by-projection sampler with theoretical approximation guarantee. The sampler also benefits exploration by coordinating agents to cover orthogonal directions in the state space during multi-agent training. We evaluate our algorithm on various cooperative benchmarks; its effectiveness has been demonstrated when compared with the state-of-the-art.

ICML 2020

References in corpus (9)

• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity
• MAVEN: Multi-Agent Variational Exploration
• Learning the Parameters of Determinantal Point Process Kernels
• A Study of AI Population Dynamics with Million-agent Reinforcement Learning

Cited by in corpus (5)

• Game-Theoretic Multiagent Reinforcement Learning
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• Settling the Variance of Multi-Agent Policy Gradients
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems