Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning

Abstract

We investigate how reinforcement learning agents can learn to cooperate. Drawing inspiration from human societies, in which successful coordination of many individuals is often facilitated by hierarchical organisation, we introduce Feudal Multi-agent Hierarchies (FMH). In this framework, a 'manager' agent, which is tasked with maximising the environmentally-determined reward function, learns to communicate subgoals to multiple, simultaneously-operating, 'worker' agents. Workers, which are rewarded for achieving managerial subgoals, take concurrent actions in the world. We outline the structure of FMH and demonstrate its potential for decentralised learning and control. We find that, given an adequate set of subgoals from which to choose, FMH performs, and particularly scales, substantially better than cooperative approaches that use a shared reward function.

References in corpus (10)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Counterfactual Multi-Agent Policy Gradients
• FeUdal Networks for Hierarchical Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning

Cited by in corpus (6)

• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Room Clearance with Feudal Hierarchical Reinforcement Learning
• Scalable, Decentralized Multi-Agent Reinforcement Learning Methods Inspired by Stigmergy and Ant Colonies