Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning

Abstract

Many tasks in artificial intelligence require the collaboration of multiple agents. We exam deep reinforcement learning for multi-agent domains. Recent research efforts often take the form of two seemingly conflicting perspectives, the decentralized perspective, where each agent is supposed to have its own controller; and the centralized perspective, where one assumes there is a larger model controlling all agents. In this regard, we revisit the idea of the master-slave architecture by incorporating both perspectives within one framework. Such a hierarchical structure naturally leverages advantages from one another. The idea of combining both perspectives is intuitive and can be well motivated from many real world systems, however, out of a variety of possible realizations, we highlights three key ingredients, i.e. composed action representation, learnable communication and independent reasoning. With network designs to facilitate these explicitly, our proposal consistently outperforms latest competing methods both in synthetic experiments and when applied to challenging StarCraft micromanagement tasks.

References in corpus (9)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Deep Reinforcement Learning: An Overview
• Learning Multiagent Communication with Backpropagation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• MazeBase: A Sandbox for Learning from Games
• ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning

Cited by in corpus (9)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Learning Attentional Communication for Multi-Agent Cooperation
• A Survey of Deep Reinforcement Learning in Video Games
• R-MADDPG for Partially Observable Environments and Limited Communication
• Multi-agent Reinforcement Learning for Networked System Control
• Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration