QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Abstract

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

Camera-ready version, International Conference of Machine Learning 2018

References in corpus (27)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Mean Field Multi-Agent Reinforcement Learning
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• The Hanabi Challenge: A New Frontier for AI Research
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Guided Deep Reinforcement Learning for Swarm Systems
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Multi-Agent Common Knowledge Reinforcement Learning
• Deep Coordination Graphs
• Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
• Deep Multi-Agent Reinforcement Learning with Relevance Graphs
• MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence
• Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces
• Action Semantics Network: Considering the Effects of Actions in Multiagent Systems
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• Stratospheric Aerosol Injection as a Deep Reinforcement Learning Problem
• Decentralized Multi-Agents by Imitation of a Centralized Controller

Cited by in corpus (76)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Federated Deep Reinforcement Learning
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Multi-Agent Common Knowledge Reinforcement Learning
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
• Deep Coordination Graphs
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Robust Multi-agent Communication via Multi-view Message Certification
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Q-value Path Decomposition for Deep Multiagent Reinforcement Learning
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning
• Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
• The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning
• Multi-Agent Game Abstraction via Graph Attention Neural Network
• Learning Individually Inferred Communication for Multi-Agent Cooperation
• MANAS: Multi-Agent Neural Architecture Search
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
• Action Semantics Network: Considering the Effects of Actions in Multiagent Systems
• On Multi-Agent Learning in Team Sports Games
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
• Macro-Action-Based Deep Multi-Agent Reinforcement Learning
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
• From Few to More: Large-scale Dynamic Multiagent Curriculum Learning
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Incorporating Pragmatic Reasoning Communication into Emergent Language
• Calibration of Shared Equilibria in General Sum Partially Observable Markov Games
• Efficient Ridesharing Dispatch Using Multi-Agent Reinforcement Learning
• Influence-Based Multi-Agent Exploration
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Model-based Multi-Agent Reinforcement Learning with Cooperative Prioritized Sweeping
• Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
• Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning
• Offline Decentralized Multi-Agent Reinforcement Learning
• Learning to Represent Action Values as a Hypergraph on the Action Vertices
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
• Off-Belief Learning
• Divergence-Regularized Multi-Agent Actor-Critic
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Decentralized Multi-Agent Actor-Critic with Generative Inference
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Net
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• Stabilizing Q Learning Via Soft Mellowmax Operator
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• Coordinated Reinforcement Learning for Optimizing Mobile Networks
• Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning
• Independent Reinforcement Learning for Weakly Cooperative Multiagent Traffic Control Problem
• Decentralized Multi-Agents by Imitation of a Centralized Controller
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• Combining Propositional Logic Based Decision Diagrams with Decision Making in Urban Systems
• Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures
• Logical Team Q-learning: An approach towards factored policies in cooperative MARL
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• Agent Probing Interaction Policies