Dota 2 with Large Scale Deep Reinforcement Learning

Abstract

On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems. OpenAI Five leveraged existing reinforcement learning techniques, scaled to learn from batches of approximately 2 million frames every 2 seconds. We developed a distributed training system and tools for continual training which allowed us to train OpenAI Five for 10 months. By defeating the Dota 2 world champion (Team OG), OpenAI Five demonstrates that self-play reinforcement learning can achieve superhuman performance on a difficult task.

References in corpus (17)

• Playing Atari with Deep Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• A Deep Reinforced Model for Abstractive Summarization
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Solving Rubik's Cube with a Robot Hand
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Exploration by Random Network Distillation
• Quantifying Generalization in Reinforcement Learning
• Emergent Complexity via Multi-Agent Competition
• Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
• An Empirical Model of Large-Batch Training
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Open-ended Learning in Symmetric Zero-sum Games
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Mix&Match - Agent Curricula for Reinforcement Learning

Cited by in corpus (264)

• Machine Learning and Deep Learning -- A review for Ecologists
• A Closer Look at Invalid Action Masking in Policy Gradient Algorithms
• A Survey of End-to-End Driving: Architectures and Training Methods
• Object-Centric Learning with Slot Attention
• A Survey of Machine Learning for Computer Architecture and Systems
• Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
• Large Language Models and the Reverse Turing Test
• A Survey of Deep Reinforcement Learning in Video Games
• The Modern Mathematics of Deep Learning
• Reinforcement Learning for Ridesharing: An Extended Survey
• Jukebox: A Generative Model for Music
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Measuring the Algorithmic Efficiency of Neural Networks
• Suphx: Mastering Mahjong with Deep Reinforcement Learning
• Exploring the landscapes of "computing": digital, neuromorphic, unconventional -- and beyond
• Decoupling Representation Learning from Reinforcement Learning
• Finding Effective Security Strategies through Reinforcement Learning and Self-Play
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Queueing Network Controls via Deep Reinforcement Learning
• A Self-adaptive SAC-PID Control Approach based on Reinforcement Learning for Mobile Robots
• Defining the scope of AI regulations
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
• Open-Ended Learning Leads to Generally Capable Agents
• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings
• Collaborative Target Search with a Visual Drone Swarm: An Adaptive Curriculum Embedded Multistage Reinforcement Learning Approach
• On the Binding Problem in Artificial Neural Networks
• Phasic Policy Gradient
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Generative Language Modeling for Automated Theorem Proving
• Modern Machine Learning Tools for Monitoring and Control of Industrial Processes: A Survey
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic
• Automated scholarly paper review: Concepts, technologies, and challenges
• Automatic Curriculum Learning through Value Disagreement
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Deep reinforcement learning applied to an assembly sequence planning problem with user preferences
• Trajectory Planning with Deep Reinforcement Learning in High-Level Action Spaces
• Learning Near-Optimal Intrusion Responses Against Dynamic Attackers
• SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
• Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents
• Generative Personas That Behave and Experience Like Humans
• Fair collaborative vehicle routing: A deep multi-agent reinforcement learning approach
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• Navigating the Landscape of Multiplayer Games
• Meta-Learning through Hebbian Plasticity in Random Networks
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• SquirRL: Automating Attack Analysis on Blockchain Incentive Mechanisms with Deep Reinforcement Learning
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Optimal Stroke Learning with Policy Gradient Approach for Robotic Table Tennis
• D-VAT: End-to-End Visual Active Tracking for Micro Aerial Vehicles
• The Artificial Intelligence behind the winning entry to the 2019 AI Robotic Racing Competition
• Multi-Agent Collaboration via Reward Attribution Decomposition
• Strong Generalization and Efficiency in Neural Programs
• An Autonomous Free Airspace En-route Controller using Deep Reinforcement Learning Techniques
• Collaborative Visual Navigation
• Artificial virtuous agents in a multiagent tragedy of the commons
• What Matters for Adversarial Imitation Learning?
• RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning
• Coordination Among Neural Modules Through a Shared Global Workspace
• Learning to Incentivize Other Learning Agents
• Reinforcement Learning for Multi-Product Multi-Node Inventory Management in Supply Chains
• Open Problems in Cooperative AI
• Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games
• Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
• Reinforcement Learning with Quantum Variational Circuits
• On the model-based stochastic value gradient for continuous reinforcement learning
• Applying Deep Reinforcement Learning to the HP Model for Protein Structure Prediction
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Heuristic-Guided Reinforcement Learning
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Revisiting Design Choices in Proximal Policy Optimization
• Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks
• Learning Generalizable Visual Representations via Interactive Gameplay
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• A Meta-Reinforcement Learning Approach to Process Control
• DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games
• Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses
• Challenges of Applying Deep Reinforcement Learning in Dynamic Dispatching
• Student-Initiated Action Advising via Advice Novelty
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Do Offline Metrics Predict Online Performance in Recommender Systems?
• Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment
• WordCraft: An Environment for Benchmarking Commonsense Agents
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• Robust Policy Gradient against Strong Data Corruption
• A Deep Reinforcement Learning Framework for Eco-driving in Connected and Automated Hybrid Electric Vehicles
• Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games
• Interpretable Learned Emergent Communication for Human-Agent Teams
• Fast Adaptation via Policy-Dynamics Value Functions
• Differentially Private Federated Learning with Laplacian Smoothing
• Human-Level Performance in No-Press Diplomacy via Equilibrium Search
• Automating Reinforcement Learning with Example-based Resets
• Reinforcement Learning Generalization with Surprise Minimization
• Provable Fictitious Play for General Mean-Field Games
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Scaling Scaling Laws with Board Games
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• SuperSuit: Simple Microwrappers for Reinforcement Learning Environments
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Evaluating the Robustness of Collaborative Agents
• Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning
• SOAC: The Soft Option Actor-Critic Architecture
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• Reinforcement Learning for Adaptive Mesh Refinement
• When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?
• HUGO -- Highlighting Unseen Grid Options: Combining Deep Reinforcement Learning with a Heuristic Target Topology Approach
• Blockchain-assisted Demonstration Cloning for Multi-Agent Deep Reinforcement Learning
• Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
• PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
• Using Unity to Help Solve Intelligence
• Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling
• Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World
• Reverb: A Framework For Experience Replay
• Finding online neural update rules by learning to remember
• The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning
• Resolving Spurious Correlations in Causal Models of Environments via Interventions
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
• Dealing with Sparse Rewards Using Graph Neural Networks
• Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature
• XDO: A Double Oracle Algorithm for Extensive-Form Games
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Convergent and Efficient Deep Q Network Algorithm
• Generalization and Robustness Implications in Object-Centric Learning
• Automated and Autonomous Experiment in Electron and Scanning Probe Microscopy
• Towards General Function Approximation in Zero-Sum Markov Games
• Heterogeneous Graph Attention Networks for Learning Diverse Communication
• Reinforcement Learning with Success Induced Task Prioritization
• Domain-Level Explainability -- A Challenge for Creating Trust in Superhuman AI Strategies
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Learning more skills through optimistic exploration
• Few Shot System Identification for Reinforcement Learning
• RL STaR Platform: Reinforcement Learning for Simulation based Training of Robots
• Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search
• Regularized Behavior Value Estimation
• Optimize Neural Fictitious Self-Play in Regret Minimization Thinking
• Taylor Expansion Policy Optimization
• Single-partition adaptive Q-learning
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• A Generalizable Approach to Learning Optimizers
• FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesis
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
• Dynamic Dispatching for Large-Scale Heterogeneous Fleet via Multi-agent Deep Reinforcement Learning
• Mutual Information State Intrinsic Control
• Reinforcement Learning Based Self-play and State Stacking Techniques for Noisy Air Combat Environment
• Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler
• Cross-Domain Imitation Learning via Optimal Transport
• Agent Environment Cycle Games
• Planning from Pixels using Inverse Dynamics Models
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Finding mixed-strategy equilibria of continuous-action games without gradients using randomized policy networks
• A survey of air combat behavior modeling using machine learning
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
• Level generation and style enhancement -- deep learning for game development overview
• Diversity-based Trajectory and Goal Selection with Hindsight Experience Replay
• Gym-$\mu$RTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Learning Equational Theorem Proving
• Policy Gradient from Demonstration and Curiosity
• A coevolutionary approach to deep multi-agent reinforcement learning
• Policy Optimization Reinforcement Learning with Entropy Regularization
• Detecting and Adapting to Novelty in Games
• Hyperparameter Transfer Across Developer Adjustments
• MimicBot: Combining Imitation and Reinforcement Learning to win in Bot Bowl
• On Proximal Policy Optimization's Heavy-tailed Gradients
• Dompteur: Taming Audio Adversarial Examples
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Continuous Control with Action Quantization from Demonstrations
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Mimicking Evolution with Reinforcement Learning
• Landmark Policy Optimization for Object Navigation Task
• Small Nash Equilibrium Certificates in Very Large Games
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Aligning an optical interferometer with beam divergence control and continuous action space
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Reinforcement Learning with Latent Flow
• Towards Interactive Reinforcement Learning with Intrinsic Feedback
• Multi-agent Cooperative Games Using Belief Map Assisted Training
• Efficient Competitive Self-Play Policy Optimization
• Mixture of Public and Private Distributions in Imperfect Information Games
• Continuous Control With Ensemble Deep Deterministic Policy Gradients
• Search in Imperfect Information Games
• DeepRepair: Style-Guided Repairing for DNNs in the Real-world Operational Environment
• Episodic Self-Imitation Learning with Hindsight
• Explainable Biomedical Recommendations via Reinforcement Learning Reasoning on Knowledge Graphs
• Strategy Game-Playing with Size-Constrained State Abstraction
• Replay-Guided Adversarial Environment Design
• Learning to Play Imperfect-Information Games by Imitating an Oracle Planner
• Towards robust and domain agnostic reinforcement learning competitions
• InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problem
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space
• Improving Multi-Agent Cooperation using Theory of Mind
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• Adaptive Agent Architecture for Real-time Human-Agent Teaming
• Neural Network Surgery with Sets
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Why is AI hard and Physics simple?
• Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios
• Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets
• Off-the-shelf deep learning is not enough: parsimony, Bayes and causality
• Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules
• Control with adaptive Q-learning
• Information theoretic analysis of computational models as a tool to understand the neural basis of behaviors
• Unsupervised Skill-Discovery and Skill-Learning in Minecraft
• ScrofaZero: Mastering Trick-taking Poker Game Gongzhu by Deep Reinforcement Learning
• Maximum Entropy Model-based Reinforcement Learning
• Reward-Free Attacks in Multi-Agent Reinforcement Learning
• VisualEnv: visual Gym environments with Blender
• Two-stage training algorithm for AI robot soccer
• Learning to Play Foosball: System and Baselines
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• Model-free Policy Learning with Reward Gradients
• Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic
• Reinforced Contact Tracing and Epidemic Intervention
• Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi
• Out-of-the-box channel pruned networks
• Explore and Control with Adversarial Surprise
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation
• Humanly Certifying Superhuman Classifiers
• Cooperative and Asynchronous Transformer-based Mission Planning for Heterogeneous Teams of Mobile Robots
• Room Clearance with Feudal Hierarchical Reinforcement Learning
• Optimizing Quantum Variational Circuits with Deep Reinforcement Learning
• Data-Efficient Methods for Dialogue Systems
• Batch size-invariance for policy optimization
• Successor Feature Neural Episodic Control
• Time-Variant Variational Transfer for Value Functions
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• Mix and Mask Actor-Critic Methods
• LBGP: Learning Based Goal Planning for Autonomous Following in Front
• Reinforcement Learning with Partial Parametric Model Knowledge
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Applying supervised and reinforcement learning methods to create neural-network-based agents for playing StarCraft II
• Hard Attention Control By Mutual Information Maximization
• HetSeq: Distributed GPU Training on Heterogeneous Infrastructure
• Deep RL Agent for a Real-Time Action Strategy Game
• Proactive and AoI-aware Failure Recovery for Stateful NFV-enabled Zero-Touch 6G Networks: Model-Free DRL Approach
• Component Transfer Learning for Deep RL Based on Abstract Representations
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Characterizing the Decidability of Finite State Automata Team Games with Communication
• High Throughput Training of Deep Surrogates from Large Ensemble Runs
• No-Press Diplomacy from Scratch