StarCraft II: A New Challenge for Reinforcement Learning

Abstract

This paper introduces SC2LE (StarCraft II Learning Environment), a reinforcement learning environment based on the StarCraft II game. This domain poses a new grand challenge for reinforcement learning, representing a more difficult class of problems than considered in most prior work. It is a multi-agent problem with multiple players interacting; there is imperfect information due to a partially observed map; it has a large action space involving the selection and control of hundreds of units; it has a large state space that must be observed solely from raw input feature planes; and it has delayed credit assignment requiring long-term strategies over thousands of steps. We describe the observation, action, and reward specification for the StarCraft II domain and provide an open source Python-based interface for communicating with the game engine. In addition to the main game maps, we provide a suite of mini-games focusing on different elements of StarCraft II gameplay. For the main game maps, we also provide an accompanying dataset of game replay data from human expert players. We give initial baseline results for neural networks trained from this data to predict game outcomes and player actions. Finally, we present initial baseline results for canonical deep reinforcement learning agents applied to the StarCraft II domain. On the mini-games, these agents learn to achieve a level of play that is comparable to a novice player. However, when trained on the main game, these agents are unable to make significant progress. Thus, SC2LE offers a new and challenging environment for exploring deep reinforcement learning algorithms and architectures.

arXiv commentCollaboration between DeepMind & Blizzard. 20 pages, 9 figures, 2 tables

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Q-learning from Demonstrations
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Move Evaluation in Go Using Deep Convolutional Neural Networks
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
• Measuring Intelligence through Games

Cited by in corpus (150)

• A Brief Survey of Deep Reinforcement Learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning for Cyber Security
• A Closer Look at Invalid Action Masking in Policy Gradient Algorithms
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• A Study on Overfitting in Deep Reinforcement Learning
• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• Relational Deep Reinforcement Learning
• Game-Theoretic Multiagent Reinforcement Learning
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings
• Multi-Agent Common Knowledge Reinforcement Learning
• Dealing with Sparse Rewards in Reinforcement Learning
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• The NetHack Learning Environment
• Extraction of Material Properties through Multi-fidelity Deep Learning from Molecular Dynamics Simulation
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
• A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue Management
• MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence
• Predicting Game Difficulty and Churn Without Players
• Hierarchical Decision Making by Generating and Following Natural Language Instructions
• Meta-learning Pseudo-differential Operators with Deep Neural Networks
• RLCard: A Toolkit for Reinforcement Learning in Card Games
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Multi-Agent Reinforcement Learning: A Report on Challenges and Approaches
• SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
• Autonomous Air Traffic Controller: A Deep Multi-Agent Reinforcement Learning Approach
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance
• Is Pessimism Provably Efficient for Offline RL?
• Network Environment Design for Autonomous Cyberdefense
• TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game
• Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
• DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters
• Leveling the Playing Field -- Fairness in AI Versus Human Game Benchmarks
• Observational Overfitting in Reinforcement Learning
• Coordination Among Neural Modules Through a Shared Global Workspace
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Generation of ice states through deep reinforcement learning
• Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
• Explaining Bayesian Neural Networks
• Optimization Methods for Interpretable Differentiable Decision Trees in Reinforcement Learning
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• Heterogeneous Multi-Agent Reinforcement Learning for Unknown Environment Mapping
• A novel policy for pre-trained Deep Reinforcement Learning for Speech Emotion Recognition
• Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
• On Multi-Agent Learning in Team Sports Games
• Flatland-RL : Multi-Agent Reinforcement Learning on Trains
• Synthesized Policies for Transfer and Adaptation across Tasks and Environments
• MSC: A Dataset for Macro-Management in StarCraft II
• Deep Reinforcement Learning with Pre-training for Time-efficient Training of Automatic Speech Recognition
• Why Build an Assistant in Minecraft?
• On the Measure of Intelligence
• Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment
• Winning Isn't Everything: Enhancing Game Development with Intelligent Agents
• Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space
• Using Fractal Neural Networks to Play SimCity 1 and Conway's Game of Life at Variable Scales
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• Decentralized Q-Learning in Zero-sum Markov Games
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• S2RMs: Spatially Structured Recurrent Modules
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
• Is Plug-in Solver Sample-Efficient for Feature-based Reinforcement Learning?
• The Natural Language of Actions
• Distal Explanations for Model-free Explainable Reinforcement Learning
• On Reinforcement Learning for Full-length Game of StarCraft
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Using Unity to Help Solve Intelligence
• Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
• Heterogeneous Learning from Demonstration
• Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications
• Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation
• ToyBox: Better Atari Environments for Testing Reinforcement Learning Agents
• Do Autonomous Agents Benefit from Hearing?
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• An Introduction of mini-AlphaStar
• Pre-training in Deep Reinforcement Learning for Automatic Speech Recognition
• Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games
• DeepCrawl: Deep Reinforcement Learning for Turn-based Strategy Games
• Neural Fictitious Self-Play on ELF Mini-RTS
• Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrations
• Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems
• A Narration-based Reward Shaping Approach using Grounded Natural Language Commands
• Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
• StarCraft II Build Order Optimization using Deep Reinforcement Learning and Monte-Carlo Tree Search
• Asynchronous Advantage Actor-Critic Agent for Starcraft II
• Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search
• The Design Of "Stratega": A General Strategy Games Framework
• Efficient Reinforcement Learning for StarCraft by Abstract Forward Models and Transfer Learning
• High-Level Strategy Selection under Partial Observability in StarCraft: Brood War
• Arena: a toolkit for Multi-Agent Reinforcement Learning
• Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space
• The Architectural Implications of Distributed Reinforcement Learning on CPU-GPU Systems
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Correcting Experience Replay for Multi-Agent Communication
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
• Energy-based Surprise Minimization for Multi-Agent Value Factorization
• Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program
• Clear the Fog: Combat Value Assessment in Incomplete Information Games with Convolutional Encoder-Decoders
• MimicBot: Combining Imitation and Reinforcement Learning to win in Bot Bowl
• Adversary agent reinforcement learning for pursuit-evasion
• Inferring Personalized Bayesian Embeddings for Learning from Heterogeneous Demonstration
• Recurrent Value Functions
• Distributed Deep Reinforcement Learning: An Overview
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• On the notion of number in humans and machines
• Gym-$\mu$RTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning
• Learning State Representations via Retracing in Reinforcement Learning
• Learning to solve arithmetic problems with a virtual abacus
• Unsupervised Hebbian Learning on Point Sets in StarCraft II
• Zero-Shot Reinforcement Learning on Graphs for Autonomous Exploration Under Uncertainty
• Dungeon Crawl Stone Soup as an Evaluation Domain for Artificial Intelligence
• Adaptive Agent Architecture for Real-time Human-Agent Teaming
• Meta Arcade: A Configurable Environment Suite for Meta-Learning
• Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules
• Neuroevolution for RTS Micro
• Exact Asymptotics for Linear Quadratic Adaptive Control
• Deep Reinforcement Learning for Playing 2.5D Fighting Games
• Autonomous Exploration Under Uncertainty via Deep Reinforcement Learning on Graphs
• Stabilizing Q Learning Via Soft Mellowmax Operator
• VisualEnv: visual Gym environments with Blender
• Growing Action Spaces
• Deep RL Agent for a Real-Time Action Strategy Game
• Playing optical tweezers with deep reinforcement learning: in virtual, physical and augmented environments
• Autonomous Industrial Management via Reinforcement Learning: Self-Learning Agents for Decision-Making -- A Review
• DinerDash Gym: A Benchmark for Policy Learning in High-Dimensional Action Space
• Assured RL: Reinforcement Learning with Almost Sure Constraints
• Amortized Variational Deep Q Network
• DeepFoldit -- A Deep Reinforcement Learning Neural Network Folding Proteins
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• GrowSpace: Learning How to Shape Plants
• Applying supervised and reinforcement learning methods to create neural-network-based agents for playing StarCraft II
• Rethinking of AlphaStar
• Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
• First Draft on the xInf Model for Universal Physical Computation and Reverse Engineering of Natural Intelligence
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning
• Minimizing Communication while Maximizing Performance in Multi-Agent Reinforcement Learning
• Deep Convolution for Irregularly Sampled Temporal Point Clouds
• BGC: Multi-Agent Group Belief with Graph Clustering
• Biased Estimates of Advantages over Path Ensembles
• Robby is Not a Robber (anymore): On the Use of Institutions for Learning Normative Behavior
• Neural Networks Models for Analyzing Magic: the Gathering Cards