Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Abstract

In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.

arXiv commentExtended version of the ICML 2018 conference paper (arXiv:1803.11485)

Cited by in corpus (28)

• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control
• RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• Enhancing the Robustness of QMIX against State-adversarial Attacks
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Adaptive trajectory-constrained exploration strategy for deep reinforcement learning
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Graph Exploration for Effective Multi-agent Q-Learning
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning
• Shared learning of powertrain control policies for vehicle fleets
• Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• Improving the Generalization of Unseen Crowd Behaviors for Reinforcement Learning based Local Motion Planners
• Logical Team Q-learning: An approach towards factored policies in cooperative MARL
• Resilient Consensus-based Multi-agent Reinforcement Learning with Function Approximation
• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning
• Two-stage training algorithm for AI robot soccer
• Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi