Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

Abstract

Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at \url{https://github.com/hijkzzz/pymarl2}.

Accepted by ICLR BlogTrack 2023

References in corpus (13)

• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• MAVEN: Multi-Agent Variational Exploration
• Phasic Policy Gradient
• Off-Policy Multi-Agent Decomposed Policy Gradients
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Macro-Action-Based Deep Multi-Agent Reinforcement Learning
• Value-Decomposition Multi-Agent Actor-Critics
• Revisiting Peng's Q($\lambda$) for Modern Reinforcement Learning

Cited by in corpus (1)

• MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning