MAVEN: Multi-Agent Variational Exploration

Abstract

Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in complex environments [43]. We specifically focus on QMIX [40], the current state-of-the-art in this domain. We show that the representational constraints on the joint action-values introduced by QMIX and similar methods lead to provably poor exploration and suboptimality. Furthermore, we propose a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control. The value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy. This allows MAVEN to achieve committed, temporally extended exploration, which is key to solving complex multi-agent tasks. Our experimental results show that MAVEN achieves significant performance improvements on the challenging SMAC domain [43].

Cited by in corpus (40)

• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• VIREL: A Variational Inference Framework for Reinforcement Learning
• Q-value Path Decomposition for Deep Multiagent Reinforcement Learning
• Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Multi-Agent Determinantal Q-Learning
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
• A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Learning more skills through optimistic exploration
• Adaptable Agent Populations via a Generative Model of Policies
• Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
• Skill Discovery of Coordination in Multi-agent Reinforcement Learning
• Model based Multi-agent Reinforcement Learning with Tensor Decompositions
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Human and Multi-Agent collaboration in a human-MARL teaming framework
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• Coordinated Proximal Policy Optimization
• Reinforcement Learning in Factored Action Spaces using Tensor Decompositions
• Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures
• Large-Scale Multi-Agent Deep FBSDEs
• DSDF: An approach to handle stochastic agents in collaborative multi-agent reinforcement learning
• Improved cooperation by balancing exploration and exploitation in intertemporal social dilemma tasks
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning