ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning

Abstract

Communication is a critical factor for the big multi-agent world to stay organized and productive. Typically, most previous multi-agent "learning-to-communicate" studies try to predefine the communication protocols or use technologies such as tabular reinforcement learning and evolutionary algorithm, which can not generalize to changing environment or large collection of agents. In this paper, we propose an Actor-Coordinator-Critic Net (ACCNet) framework for solving "learning-to-communicate" problem. The ACCNet naturally combines the powerful actor-critic reinforcement learning technology with deep learning technology. It can efficiently learn the communication protocols even from scratch under partially observable environment. We demonstrate that the ACCNet can achieve better results than several baselines under both continuous and discrete action space environments. We also analyse the learned protocols and discuss some design considerations.

V3 of original submission. Actor-Critic Method for Multi-agent Learning-to-Communicate based on Deep Reinforcement Learning, It is suitable for both continuous and discrete action space environments

References in corpus (13)

• Continuous control with deep reinforcement learning
• Prioritized Experience Replay
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Decentralized Control of Cooperative Systems: Categorization and Complexity Analysis
• Winner-Take-All Autoencoders
• k-Sparse Autoencoders
• Deep Attention Recurrent Q-Network
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
• MazeBase: A Sandbox for Learning from Games
• Fully Decentralized Policies for Multi-Agent Systems: An Information Theoretic Approach
• Collaborative Deep Reinforcement Learning for Joint Object Search

Cited by in corpus (10)

• Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG
• Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
• Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing
• Model-based Reinforcement Learning for Service Mesh Fault Resiliency in a Web Application-level
• Learning Multi-agent Communication under Limited-bandwidth Restriction for Internet Packet Routing
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• Learning to Communicate Using Counterfactual Reasoning
• Event-Triggered Multi-agent Reinforcement Learning with Communication under Limited-bandwidth Constraint
• Stigmergic Independent Reinforcement Learning for Multi-Agent Collaboration
• SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition