Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment

Abstract

Extending transfer learning to cooperative multi-agent reinforcement learning (MARL) has recently received much attention. In contrast to the single-agent setting, the coordination indispensable in cooperative MARL constrains each agent's policy. However, existing transfer methods focus exclusively on agent policy and ignores coordination knowledge. We propose a new architecture that realizes robust coordination knowledge transfer through appropriate decomposition of the overall coordination into several coordination patterns. We use a novel mixing network named level-adaptive QTransformer (LA-QTransformer) to realize agent coordination that considers credit assignment, with appropriate coordination patterns for different agents realized by a novel level-adaptive Transformer (LA-Transformer) dedicated to the transfer of coordination knowledge. In addition, we use a novel agent network named Population Invariant agent with Transformer (PIT) to realize the coordination transfer in more varieties of scenarios. Extensive experiments in StarCraft II micro-management show that LA-QTransformer together with PIT achieves superior performance compared with state-of-the-art baselines.

arXiv comment12 pages, 9 figures

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Categorical Reparameterization with Gumbel-Softmax
• Dota 2 with Large Scale Deep Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• StarCraft II: A New Challenge for Reinforcement Learning
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Deep Reinforcement Learning for List-wise Recommendations
• Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
• Off-Policy Multi-Agent Decomposed Policy Gradients
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• Action Semantics Network: Considering the Effects of Actions in Multiagent Systems