Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games

Abstract

Many artificial intelligence (AI) applications often require multiple intelligent agents to work in a collaborative effort. Efficient learning for intra-agent communication and coordination is an indispensable step towards general AI. In this paper, we take StarCraft combat game as a case study, where the task is to coordinate multiple agents as a team to defeat their enemies. To maintain a scalable yet effective communication protocol, we introduce a Multiagent Bidirectionally-Coordinated Network (BiCNet ['bIknet]) with a vectorised extension of actor-critic formulation. We show that BiCNet can handle different types of combats with arbitrary numbers of AI agents for both sides. Our analysis demonstrates that without any supervisions such as human demonstrations or labelled data, BiCNet could learn various types of advanced coordination strategies that have been commonly used by experienced game players. In our experiments, we evaluate our approach against multiple baselines under different scenarios; it shows state-of-the-art performance, and possesses potential values for large-scale real-world applications.

10 pages, 10 figures. Previously as title: "Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games", Mar 2017

References in corpus (6)

• Deep Learning in Neural Networks: An Overview
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• StarCraft II: A New Challenge for Reinforcement Learning
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games

Cited by in corpus (83)

• A Brief Survey of Deep Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Mean Field Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning
• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
• R-MADDPG for Partially Observable Environments and Limited Communication
• Multi-agent Reinforcement Learning for Networked System Control
• Learning to Schedule Communication in Multi-agent Reinforcement Learning
• UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios
• Multi-Agent Common Knowledge Reinforcement Learning
• Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
• Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks
• ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• Graph Convolutional Reinforcement Learning
• VAIN: Attentional Multi-agent Predictive Modeling
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• STARDATA: A StarCraft AI Research Dataset
• SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• TarMAC: Targeted Multi-Agent Communication
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Settling the Variance of Multi-Agent Policy Gradients
• A Generalized Training Approach for Multiagent Learning
• Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication
• Multi-agent Deep Reinforcement Learning with Extremely Noisy Observations
• Learning Individually Inferred Communication for Multi-Agent Cooperation
• Collaborative Visual Navigation
• Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
• Depthwise Convolution for Multi-Agent Communication with Enhanced Mean-Field Approximation
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• M${}^3$RL: Mind-aware Multi-agent Management Reinforcement Learning
• MSC: A Dataset for Macro-Management in StarCraft II
• Learning Multi-agent Communication under Limited-bandwidth Restriction for Internet Packet Routing
• Learning in Nonzero-Sum Stochastic Games with Potentials
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Towards Heterogeneous Multi-Agent Reinforcement Learning with Graph Neural Networks
• Multi-Agent Trust Region Policy Optimization
• Com-DDPG: A Multiagent Reinforcement Learning-based Offloading Strategy for Mobile Edge Computing
• Multi-Agent Actor-Critic with Hierarchical Graph Attention Network
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance
• Multi-Agent Determinantal Q-Learning
• When2com: Multi-Agent Perception via Communication Graph Grouping
• Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation
• Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning
• Event-Triggered Multi-agent Reinforcement Learning with Communication under Limited-bandwidth Constraint
• Value-Decomposition Multi-Agent Actor-Critics
• MAIDCRL: Semi-centralized Multi-Agent Influence Dense-CNN Reinforcement Learning
• Efficient Ridesharing Dispatch Using Multi-Agent Reinforcement Learning
• StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
• Monte Carlo Neural Fictitious Self-Play: Approach to Approximate Nash equilibrium of Imperfect-Information Games
• Heterogeneous Graph Attention Networks for Learning Diverse Communication
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems
• An Actor-Critic-Attention Mechanism for Deep Reinforcement Learning in Multi-view Environments
• Correcting Experience Replay for Multi-Agent Communication
• Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning
• Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learning
• Bi-level Actor-Critic for Multi-agent Coordination
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• Who2com: Collaborative Perception via Learnable Handshake Communication
• Active Perception in Adversarial Scenarios using Maximum Entropy Deep Reinforcement Learning
• AB-Mapper: Attention and BicNet Based Multi-agent Path Finding for Dynamic Crowded Environment
• KnowSR: Knowledge Sharing among Homogeneous Agents in Multi-agent Reinforcement Learning
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• Efficient Policy Learning for Non-Stationary MDPs under Adversarial Manipulation
• Multi-agent Collaboration for Feasible Collaborative Behavior Construction and Evaluation
• Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep Reinforcement Learning
• Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
• Learning to Shape Rewards using a Game of Two Partners
• Iterative Update and Unified Representation for Multi-Agent Reinforcement Learning
• Learning to Coordinate via Multiple Graph Neural Networks