QPLEX: Duplex Dueling Multi-Agent Q-Learning

Abstract

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

References in corpus (5)

• Challenges of Real-World Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• MAVEN: Multi-Agent Variational Exploration
• Off-Policy Multi-Agent Decomposed Policy Gradients

Cited by in corpus (23)

• RODE: Learning Roles to Decompose Multi-Agent Tasks
• UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Context-Aware Sparse Deep Coordination Graphs
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Learning Long-Term Reward Redistribution via Randomized Return Decomposition
• Novel Multi-Agent Action Masked Deep Reinforcement Learning for General Industrial Assembly Lines Balancing Problems
• Model based Multi-agent Reinforcement Learning with Tensor Decompositions
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Divergence-Regularized Multi-Agent Actor-Critic
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning