FACMAC: Factored Multi-Agent Centralised Policy Gradients

Abstract

We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. Like MADDPG, a popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns a centralised but factored critic, which combines per-agent utilities into the joint action-value function via a non-linear monotonic function, as in QMIX, a popular multi-agent Q-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics. In addition, FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains.

References in corpus (11)

• StarCraft II: A New Challenge for Reinforcement Learning
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Parameter Space Noise for Exploration
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
• Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics
• R-MADDPG for Partially Observable Environments and Limited Communication
• Deep Coordination Graphs
• Q-Learning in enormous action spaces via amortized approximate maximization
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning

Cited by in corpus (13)

• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Settling the Variance of Multi-Agent Policy Gradients
• Multi-Agent Constrained Policy Optimisation
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Learning in Nonzero-Sum Stochastic Games with Potentials
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• Formal Modelling for Multi-Robot Systems Under Uncertainty
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Offline Decentralized Multi-Agent Reinforcement Learning
• Novel Multi-Agent Action Masked Deep Reinforcement Learning for General Industrial Assembly Lines Balancing Problems
• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning