QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning

Abstract

QTRAN is a multi-agent reinforcement learning (MARL) algorithm capable of learning the largest class of joint-action value functions up to date. However, despite its strong theoretical guarantee, it has shown poor empirical performance in complex environments, such as Starcraft Multi-Agent Challenge (SMAC). In this paper, we identify the performance bottleneck of QTRAN and propose a substantially improved version, coined QTRAN++. Our gains come from (i) stabilizing the training objective of QTRAN, (ii) removing the strict role separation between the action-value estimators of QTRAN, and (iii) introducing a multi-head mixing network for value transformation. Through extensive evaluation, we confirm that our diagnosis is correct, and QTRAN++ successfully bridges the gap between empirical performance and theoretical guarantee. In particular, QTRAN++ newly achieves state-of-the-art performance in the SMAC environment. The code will be released.

References in corpus (10)

• Continuous control with deep reinforcement learning
• Counterfactual Multi-Agent Policy Gradients
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
• MAVEN: Multi-Agent Variational Exploration
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Multi-Agent Determinantal Q-Learning

Cited by in corpus (4)

• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients