Playing FPS Games with Deep Reinforcement Learning

Abstract

Advances in deep reinforcement learning have allowed autonomous agents to perform well on Atari games, often outperforming humans, using only raw pixels to make their decisions. However, most of these games take place in 2D environments that are fully observable to the agent. In this paper, we present the first architecture to tackle 3D environments in first-person shooter games, that involve partially observable states. Typically, deep reinforcement learning methods only utilize visual input for training. We present a method to augment these models to exploit game feature information such as the presence of enemies or items, during the training phase. Our model is trained to simultaneously learn these features along with minimizing a Q-learning objective, which is shown to dramatically improve the training speed and performance of our agent. Our architecture is also modularized to allow different models to be independently trained for different phases of the game. We show that the proposed architecture substantially outperforms built-in AI agents of the game as well as humans in deathmatch scenarios.

The authors contributed equally to this work

Cited by in corpus (42)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Unity: A General Platform for Intelligent Agents
• Learning Attentional Communication for Multi-Agent Cooperation
• Object Goal Navigation using Goal-Oriented Semantic Exploration
• Review: Deep Learning in Electron Microscopy
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings
• Learning to Navigate in Cities Without a Map
• The StreetLearn Environment and Dataset
• DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning
• Predicting Game Difficulty and Churn Without Players
• Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge
• Mastering Complex Control in MOBA Games with Deep Reinforcement Learning
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Deep Counterfactual Regret Minimization
• Shaping Belief States with Generative Environment Models for RL
• Navigating Assistance System for Quadcopter with Deep Reinforcement Learning
• Recurrent Off-policy Baselines for Memory-based Continuous Control
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Learning Generalizable Visual Representations via Interactive Gameplay
• An Introduction to Neural Architecture Search for Convolutional Networks
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• A Q-values Sharing Framework for Multiagent Reinforcement Learning under Budget Constraint
• MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning
• Hierarchical principles of embodied reinforcement learning: A review
• Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search
• Off-Policy Actor-Critic with Shared Experience Replay
• Zero-Shot Learning of Text Adventure Games with Sentence-Level Semantics
• Evolving Neural Networks in Reinforcement Learning by means of UMDAc
• Co-training for Policy Learning
• Influence-aware Memory Architectures for Deep Reinforcement Learning
• Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships
• Discovering Differential Features: Adversarial Learning for Information Credibility Evaluation
• Explanation of Reinforcement Learning Model in Dynamic Multi-Agent System
• C-3PO: Cyclic-Three-Phase Optimization for Human-Robot Motion Retargeting based on Reinforcement Learning
• Compare and Select: Video Summarization with Multi-Agent Reinforcement Learning
• Simultaneous Navigation and Construction Benchmarking Environments
• Towards Brain-inspired System: Deep Recurrent Reinforcement Learning for Simulated Self-driving Agent
• Lifetime policy reuse and the importance of task capacity
• Expert Human-Level Driving in Gran Turismo Sport Using Deep Reinforcement Learning with Image-based Representation
• Building Intelligent Autonomous Navigation Agents
• Augmenting Automated Game Testing with Deep Reinforcement Learning