Value-Decomposition Multi-Agent Actor-Critics

Abstract

The exploitation of extra state information has been an active research area in multi-agent reinforcement learning (MARL). QMIX represents the joint action-value using a non-negative function approximator and achieves the best performance, by far, on multi-agent benchmarks, StarCraft II micromanagement tasks. However, our experiments show that, in some cases, QMIX is incompatible with A2C, a training paradigm that promotes algorithm training efficiency. To obtain a reasonable trade-off between training efficiency and algorithm performance, we extend value-decomposition to actor-critics that are compatible with A2C and propose a novel actor-critic framework, value-decomposition actor-critics (VDACs). We evaluate VDACs on the testbed of StarCraft II micromanagement tasks and demonstrate that the proposed framework improves median performance over other actor-critic methods. Furthermore, we use a set of ablation experiments to identify the key factors that contribute to the performance of VDACs.

Accepted by 35th AAAI Conference on Artificial Intelligence (AAAI 2021)

References in corpus (13)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Continuous control with deep reinforcement learning
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Multiagent Soft Q-Learning
• Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication

Cited by in corpus (1)

• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning