Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

Abstract

In many real-world tasks, multiple agents must learn to coordinate with each other given their private observations and limited communication ability. Deep multiagent reinforcement learning (Deep-MARL) algorithms have shown superior performance in such challenging settings. One representative class of work is multiagent value decomposition, which decomposes the global shared multiagent Q-value $Q_{tot}$ into individual Q-values $Q^i$ to guide individuals' behaviors, i.e. VDN imposing an additive formation and QMIX adopting a monotonic assumption using an implicit mixing method. However, most of the previous efforts impose certain assumptions between $Q_{tot}$ and $Q^i$ and lack theoretical groundings. Besides, they do not explicitly consider the agent-level impact of individuals to the whole system when transforming individual $Q^i$s into $Q_{tot}$. In this paper, we theoretically derive a general formula of $Q_{tot}$ in terms of $Q^i$, based on which we can naturally implement a multi-head attention formation to approximate $Q_{tot}$, resulting in not only a refined representation of $Q_{tot}$ with an agent-level attention mechanism, but also a tractable maximization algorithm of decentralized policies. Extensive experiments demonstrate that our method outperforms state-of-the-art MARL methods on the widely adopted StarCraft benchmark across different scenarios, and attention analysis is further conducted with valuable insights.

References in corpus (3)

• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Control of Memory, Active Perception, and Action in Minecraft
• Are Transformers universal approximators of sequence-to-sequence functions?

Cited by in corpus (20)

• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios
• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
• Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Divergence-Regularized Multi-Agent Actor-Critic
• Reinforcement Learning in Factored Action Spaces using Tensor Decompositions
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients