HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

Abstract

Recently, some challenging tasks in multi-agent systems have been solved by some hierarchical reinforcement learning methods. Inspired by the intra-level and inter-level coordination in the human nervous system, we propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for fully cooperative multi-agent problems. To address the instability arising from the concurrent optimization of policies between various levels and agents, we introduce the dual coordination mechanism of inter-level and inter-agent strategies by designing reward functions in a two-level hierarchy. HAVEN does not require domain knowledge and pre-training, and can be applied to any value decomposition variant. Our method achieves desirable results on different decentralized partially observable Markov decision process domains and outperforms other popular multi-agent hierarchical reinforcement learning algorithms.

arXiv comment13 pages, 11 figures, 2 tables

References in corpus (15)

• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Learning Multiagent Communication with Backpropagation
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• MAVEN: Multi-Agent Variational Exploration
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• Inter-Level Cooperation in Hierarchical Reinforcement Learning