Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
arXiv:1609.02993
Abstract
We consider scenarios from the real-time strategy game StarCraft as new benchmarks for reinforcement learning algorithms. We propose micromanagement tasks, which present the problem of the short-term, low-level control of army members during a battle. From a reinforcement learning point of view, these scenarios are challenging because the state-action space is very large, and because there is no obvious feature representation for the state-action evaluation function. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features given by the game engine. In addition, we present a heuristic reinforcement learning algorithm which combines direct exploration in the policy space and backpropagation. This algorithm allows for the collection of traces for learning using deterministic policies, which appears much more efficient than, for example, ε-greedy exploration. Experiments show that with this algorithm, we successfully learn non-trivial strategies for scenarios with armies of up to 15 agents, where both Q-learning and REINFORCE struggle.
18 pages, 1 figure (2 plots), 2 tables
References in corpus (1)
Cited by in corpus (43)
- A Brief Survey of Deep Reinforcement Learning
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Deep Reinforcement Learning: An Overview
- Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
- Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
- Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
- A Survey of Deep Reinforcement Learning in Video Games
- Maintaining cooperation in complex social dilemmas using deep reinforcement learning
- TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
- ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games
- TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
- Multi-Agent Common Knowledge Reinforcement Learning
- Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
- Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks
- Hierarchical Decision Making by Generating and Following Natural Language Instructions
- ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
- VAIN: Attentional Multi-agent Predictive Modeling
- STARDATA: A StarCraft AI Research Dataset
- SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
- Learning Nearly Decomposable Value Functions Via Communication Minimization
- TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game
- Combining Strategic Learning and Tactical Search in Real-Time Strategy Games
- Consequentialist conditional cooperation in social dilemmas with imperfect information
- A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges
- Value Propagation Networks
- Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
- On Reinforcement Learning for Full-length Game of StarCraft
- Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications
- An Introduction of mini-AlphaStar
- Value-Decomposition Multi-Agent Actor-Critics
- StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
- Clear the Fog: Combat Value Assessment in Incomplete Information Games with Convolutional Encoder-Decoders
- DeepConfig: Automating Data Center Network Topologies Management with Machine Learning
- Defining Digital Quadruplets in the Cyber-Physical-Social Space for Parallel Driving
- Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games
- Learning Transition Models with Time-delayed Causal Relations
- Decentralized Multi-Agents by Imitation of a Centralized Controller
- FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
- Combinatorial Multi-armed Bandits for Real-Time Strategy Games
- Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
- Growing Action Spaces