TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game

Abstract

StarCraft, one of the most difficult esport games with long-standing history of professional tournaments, has attracted generations of players and fans, and also, intense attentions in artificial intelligence research. Recently, Google's DeepMind announced AlphaStar, a grandmaster level AI in StarCraft II that can play with humans using comparable action space and operations. In this paper, we introduce a new AI agent, named TStarBot-X, that is trained under orders of less computations and can play competitively with expert human players. TStarBot-X takes advantage of important techniques introduced in AlphaStar, and also benefits from substantial innovations including new league training methods, novel multi-agent roles, rule-guided policy search, stabilized policy improvement, lightweight neural network architecture, and importance sampling in imitation learning, etc. We show that with orders of less computation scale, a faithful reimplementation of AlphaStar's methods can not succeed and the proposed techniques are necessary to ensure TStarBot-X's competitive performance. We reveal all technical details that are complementary to those mentioned in AlphaStar, showing the most sensitive parts in league training, reinforcement learning and imitation learning that affect the performance of the agents. Most importantly, this is an open-sourced study that all codes and resources (including the trained model parameters) are publicly accessible via \url{https://github.com/tencent-ailab/tleague_projpage}. We expect this study could be beneficial for both academic and industrial future research in solving complex problems like StarCraft, and also, might provide a sparring partner for all StarCraft II players and other AI agents.

arXiv comment29 pages

References in corpus (11)

• Trust Region Policy Optimization
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Actor-Critic Policy Optimization in Partially Observable Multiagent Environments
• TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
• Open-ended Learning in Symmetric Zero-sum Games
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• StarCraft II Build Order Optimization using Deep Reinforcement Learning and Monte-Carlo Tree Search
• Arena: a toolkit for Multi-Agent Reinforcement Learning

Cited by in corpus (7)

• Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
• SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• An Empirical Study on Google Research Football Multi-agent Scenarios
• Gym-$\mu$RTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning
• Applying supervised and reinforcement learning methods to create neural-network-based agents for playing StarCraft II