StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning

Abstract

Real-time strategy games have been an important field of game artificial intelligence in recent years. This paper presents a reinforcement learning and curriculum transfer learning method to control multiple units in StarCraft micromanagement. We define an efficient state representation, which breaks down the complexity caused by the large state space in the game environment. Then a parameter sharing multi-agent gradientdescent Sarsa(λ) (PS-MAGDS) algorithm is proposed to train the units. The learning policy is shared among our units to encourage cooperative behaviors. We use a neural network as a function approximator to estimate the action-value function, and propose a reward function to help units balance their move and attack. In addition, a transfer learning method is used to extend our model to more difficult scenarios, which accelerates the training process and improves the learning performance. In small scale scenarios, our units successfully learn to combat and defeat the built-in AI with 100% win rates. In large scale scenarios, curriculum transfer learning method is used to progressively train a group of units, and shows superior performance over some baseline methods in target scenarios. With reinforcement learning and curriculum transfer learning, our units are able to learn appropriate strategies in StarCraft micromanagement scenarios.

arXiv comment12 pages, 14 figures, accepted to IEEE Transactions on Emerging Topics in Computational Intelligence

References in corpus (11)

• Continuous control with deep reinforcement learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Counterfactual Multi-Agent Policy Gradients
• Massively Parallel Methods for Deep Reinforcement Learning
• Learning to reinforcement learn
• Learning to Navigate in Complex Environments
• Learning Multiagent Communication with Backpropagation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks