Multi-Agent Collaboration via Reward Attribution Decomposition

Abstract

Recent advances in multi-agent reinforcement learning (MARL) have achieved super-human performance in games like Quake 3 and Dota 2. Unfortunately, these techniques require orders-of-magnitude more training rounds than humans and don't generalize to new agent configurations even on the same game. In this work, we propose Collaborative Q-learning (CollaQ) that achieves state-of-the-art performance in the StarCraft multi-agent challenge and supports ad hoc team play. We first formulate multi-agent collaboration as a joint optimization on reward assignment and show that each agent has an approximately optimal policy that decomposes into two parts: one part that only relies on the agent's own state, and the other part that is related to states of nearby agents. Following this novel finding, CollaQ decomposes the Q-function of each agent into a self term and an interactive term, with a Multi-Agent Reward Attribution (MARA) loss that regularizes the training. CollaQ is evaluated on various StarCraft maps and shows that it outperforms existing state-of-the-art techniques (i.e., QMIX, QTRAN, and VDN) by improving the win rate by 40% with the same number of samples. In the more challenging ad hoc team play setting (i.e., reweight/add/remove units without re-training or finetuning), CollaQ outperforms previous SoTA by over 30%.

References in corpus (13)

• Playing Atari with Deep Reinforcement Learning
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• Emergent Tool Use From Multi-Agent Autocurricula
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• "Other-Play" for Zero-Shot Coordination
• Finding Friend and Foe in Multi-Agent Games
• A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning
• Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

Cited by in corpus (5)

• Spatial Intention Maps for Multi-Agent Mobile Manipulation
• Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• Learning cooperative behaviours in adversarial multi-agent systems