Human-level performance in first-person multiplayer games with population-based deep reinforcement learning

Abstract

Recent progress in artificial intelligence through reinforcement learning (RL) has shown great success on increasingly complex single-agent environments and two-player turn-based games. However, the real-world contains multiple agents, each learning and acting independently to cooperate and compete with other agents, and environments reflecting this degree of complexity remain an open challenge. In this work, we demonstrate for the first time that an agent can achieve human-level in a popular 3D multiplayer first-person video game, Quake III Arena Capture the Flag, using only pixels and game points as input. These results were achieved by a novel two-tier optimisation process in which a population of independent RL agents are trained concurrently from thousands of parallel matches with agents playing in teams together and against each other on randomly generated environments. Each agent in the population learns its own internal reward signal to complement the sparse delayed reward from winning, and selects actions using a novel temporally hierarchical representation that enables the agent to reason at multiple timescales. During game-play, these agents display human-like behaviours such as navigating, following, and defending based on a rich learned representation that is shown to encode high-level game knowledge. In an extensive tournament-style evaluation the trained agents exceeded the win-rate of strong human players both as teammates and opponents, and proved far stronger than existing state-of-the-art agents. These results demonstrate a significant jump in the capabilities of artificial agents, bringing us closer to the goal of human-level intelligence.

References in corpus (16)

• Continuous control with deep reinforcement learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Recurrent Latent Variable Model for Sequential Data
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• A Clockwork RNN
• Learning Multiagent Communication with Backpropagation
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• FeUdal Networks for Hierarchical Reinforcement Learning
• The Complexity of Decentralized Control of Markov Decision Processes
• Sequential Neural Models with Stochastic Layers
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
• Deep Reinforcement Learning from Self-Play in Imperfect-Information Games

Cited by in corpus (152)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning for Cyber Security
• Emergent Tool Use From Multi-Agent Autocurricula
• Compute Trends Across Three Eras of Machine Learning
• The Hanabi Challenge: A New Frontier for AI Research
• From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
• Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning
• Game-Theoretic Multiagent Reinforcement Learning
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• Optical coherent dot-product chip for sophisticated deep learning regression
• Scalable agent alignment via reward modeling: a research direction
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• Evolutionary Reinforcement Learning: A Survey
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Computer-inspired Quantum Experiments
• A Survey on Reinforcement Learning for Recommender Systems
• Hierarchical visuomotor control of humanoids
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Deep Neural Networks using a Single Neuron: Folded-in-Time Architecture using Feedback-Modulated Delay Loops
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• From Chess and Atari to StarCraft and Beyond: How Game AI is Driving the World of AI
• Neurosymbolic Reinforcement Learning and Planning: A Survey
• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings
• Quantum optimal control of multi-level dissipative quantum systems with Reinforcement Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents
• Open-ended Learning in Symmetric Zero-sum Games
• Reinforcement Learning Applications
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• Reward Shaping via Meta-Learning
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Derivative-Free Reinforcement Learning: A Review
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• A social path to human-like artificial intelligence
• AI in Human-computer Gaming: Techniques, Challenges and Opportunities
• Automated curricula through setter-solver interactions
• Mind the gap: Challenges of deep learning approaches to Theory of Mind
• Spatial Intention Maps for Multi-Agent Mobile Manipulation
• TorchBeast: A PyTorch Platform for Distributed RL
• Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games
• MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning
• Mastering Complex Control in MOBA Games with Deep Reinforcement Learning
• TarMAC: Targeted Multi-Agent Communication
• Meta-trained agents implement Bayes-optimal agents
• A community-powered search of machine learning strategy space to find NMR property prediction models
• TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game
• A Generalized Training Approach for Multiagent Learning
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Mimetic Models: Ethical Implications of AI that Acts Like You
• GPT for Games: A Scoping Review (2020-2023)
• Learning First-to-Spike Policies for Neuromorphic Control Using Policy Gradients
• Quantifying the effects of environment and population diversity in multi-agent reinforcement learning
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• Collaborative Visual Navigation
• Finding Friend and Foe in Multi-Agent Games
• Learning to Incentivize Other Learning Agents
• Open Problems in Cooperative AI
• Ease-of-Teaching and Language Structure from Emergent Communication
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• SC2EGSet: StarCraft II Esport Replay and Game-state Dataset
• Development of collective behavior in newborn artificial agents
• Towards More Human-like AI Communication: A Review of Emergent Communication Research
• Learning Latent Representations to Co-Adapt to Humans
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Energy-Based Imitation Learning
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Soft Hindsight Experience Replay
• Learning in Nonzero-Sum Stochastic Games with Potentials
• ORRB -- OpenAI Remote Rendering Backend
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Automated System-level Testing of Unmanned Aerial Systems
• Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games
• Real World Games Look Like Spinning Tops
• Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Games
• Fast Adaptation via Policy-Dynamics Value Functions
• MULEX: Disentangling Exploitation from Exploration in Deep RL
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games
• Evaluating the Robustness of Collaborative Agents
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Long-Term Planning and Situational Awareness in OpenAI Five
• Scaling Imitation Learning in Minecraft
• Human AI interaction loop training: New approach for interactive reinforcement learning
• Using Unity to Help Solve Intelligence
• Collective Intelligence for Deep Learning: A Survey of Recent Developments
• Continuous Control with Deep Reinforcement Learning for Autonomous Vessels
• Population-Guided Parallel Policy Search for Reinforcement Learning
• Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling
• Cooperative Coevolution for Non-Separable Large-Scale Black-Box Optimization: Convergence Analyses and Distributed Accelerations
• There is no Artificial General Intelligence
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• XDO: A Double Oracle Algorithm for Extensive-Form Games
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search
• Faster Improvement Rate Population Based Training
• Coordination in Adversarial Sequential Team Games via Multi-Agent Deep Reinforcement Learning
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Phoebe: Reuse-Aware Online Caching with Reinforcement Learning for Emerging Storage Models
• Adaptive Synthetic Characters for Military Training
• Arena: a toolkit for Multi-Agent Reinforcement Learning
• A novel method for identifying the deep neural network model with the Serial Number
• A Conceptual Bio-Inspired Framework for the Evolution of Artificial General Intelligence
• Influence-aware Memory Architectures for Deep Reinforcement Learning
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers
• Experimental semi-autonomous eigensolver using reinforcement learning
• Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication
• Level generation and style enhancement -- deep learning for game development overview
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesis
• Player-AI Interaction: What Neural Network Games Reveal About AI as Play
• Multi-Agent Coordination in Adversarial Environments through Signal Mediated Strategies
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Bridging the gap between emotion and joint action
• Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation
• Detecting and Adapting to Novelty in Games
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Training Learned Optimizers with Randomly Initialized Learned Optimizers
• Performance evaluation of an integrated photonic convolutional neural network based on delay buffering and wavelength division multiplexing
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Efficient Competitive Self-Play Policy Optimization
• Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledge
• Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networks
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Benchmarking Population-Based Reinforcement Learning across Robotic Tasks with GPU-Accelerated Simulation
• Fast Task-Adaptation for Tasks Labeled Using Natural Language in Reinforcement Learning
• Exploiting Language Instructions for Interpretable and Compositional Reinforcement Learning
• Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules
• Playing optical tweezers with deep reinforcement learning: in virtual, physical and augmented environments
• Iterative Update and Unified Representation for Multi-Agent Reinforcement Learning
• Simultaneous Navigation and Construction Benchmarking Environments
• Neural Network Verification for the Masses (of AI graduates)
• Negotiating Team Formation Using Deep Reinforcement Learning
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Learning to Win, Lose and Cooperate through Reward Signal Evolution
• A Benchmarking Framework for Interactive 3D Applications in the Cloud
• Universal Policies to Learn Them All