Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

Abstract

The game of chess is the most widely-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. In contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go, by tabula rasa reinforcement learning from games of self-play. In this paper, we generalise this approach into a single AlphaZero algorithm that can achieve, tabula rasa, superhuman performance in many challenging domains. Starting from random play, and given no domain knowledge except the game rules, AlphaZero achieved within 24 hours a superhuman level of play in the games of chess and shogi (Japanese chess) as well as Go, and convincingly defeated a world-champion program in each case.

References in corpus (4)

• Thinking Fast and Slow with Deep Learning and Tree Search
• Move Evaluation in Go Using Deep Convolutional Neural Networks
• Giraffe: Using Deep Reinforcement Learning to Play Chess
• Understanding Sampling Style Adversarial Search Methods

Cited by in corpus (266)

• Machine learning-assisted directed protein evolution with combinatorial libraries
• Solving Rubik's Cube with a Robot Hand
• Artificial Intelligence and Big Data in Entrepreneurship: A New Era Has Begun
• Large-Scale Optical Neural Networks based on Photoelectric Multiplication
• Emergent Tool Use From Multi-Agent Autocurricula
• A Survey of End-to-End Driving: Architectures and Training Methods
• Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• The Hanabi Challenge: A New Frontier for AI Research
• A Study on Overfitting in Deep Reinforcement Learning
• Adversarial Examples: Attacks and Defenses for Deep Learning
• Ablation Studies in Artificial Neural Networks
• Building a Conversational Agent Overnight with Dialogue Self-Play
• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• Meta-modeling game for deriving theoretical-consistent, micro-structural-based traction-separation laws via deep reinforcement learning
• Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
• Harms from Increasingly Agentic Algorithmic Systems
• RUDDER: Return Decomposition for Delayed Rewards
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Innateness, AlphaZero, and Artificial Intelligence
• ViZDoom Competitions: Playing Doom from Pixels
• Programmatically Interpretable Reinforcement Learning
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Digital Twin: Values, Challenges and Enablers
• DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM
• A newcomer's guide to deep learning for inverse design in nano-photonics
• Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States
• Neurosymbolic Reinforcement Learning and Planning: A Survey
• First Order Constrained Optimization in Policy Space
• On the Binding Problem in Artificial Neural Networks
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Breaking Adiabatic Quantum Control with Deep Learning
• Modified DDPG car-following model with a real-world human driving experience with CARLA simulator
• Adaptive Optics control using Model-Based Reinforcement Learning
• Manifold learning for parameter reduction
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Generative Language Modeling for Automated Theorem Proving
• Coloring Big Graphs with AlphaGoZero
• A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks
• Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimization
• Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving
• Low-Memory Neural Network Training: A Technical Report
• Re-evaluating Evaluation
• Making Deep Q-learning methods robust to time discretization
• Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence
• Uncovering Instabilities in Variational-Quantum Deep Q-Networks
• Parameterized Reinforcement Learning for Optical System Optimization
• A non-cooperative meta-modeling game for automated third-party calibrating, validating, and falsifying constitutive laws with parallelized adversarial attacks
• AI safety via debate
• Deep Learning Based Simulators for the Phosphorus Removal Process Control in Wastewater Treatment via Deep Reinforcement Learning Algorithms
• Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents
• Supervising strong learners by amplifying weak experts
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• Mastering Complex Control in MOBA Games with Deep Reinforcement Learning
• How Much Automation Does a Data Scientist Want?
• Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019
• Multi-task Deep Reinforcement Learning with PopArt
• Solving the single-track train scheduling problem via Deep Reinforcement Learning
• Backplay: "Man muss immer umkehren"
• The Chess Transformer: Mastering Play using Generative Language Models
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• AlphaD3M: Machine Learning Pipeline Synthesis
• Deep Counterfactual Regret Minimization
• SquirRL: Automating Attack Analysis on Blockchain Incentive Mechanisms with Deep Reinforcement Learning
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• A0C: Alpha Zero in Continuous Action Space
• Leveling the Playing Field -- Fairness in AI Versus Human Game Benchmarks
• Beyond the One Step Greedy Approach in Reinforcement Learning
• Solving the Rubik's Cube Without Human Knowledge
• Exploring search space trees using an adapted version of Monte Carlo tree search for combinatorial optimization problems
• Policy Gradient Search: Online Planning and Expert Iteration without Search Trees
• Open Problems in Cooperative AI
• Planning With Uncertain Specifications (PUnS)
• Defending Against Adversarial Examples with K-Nearest Neighbor
• Harnessing Structures for Value-Based Planning and Reinforcement Learning
• Scratch that! An Evolution-based Adversarial Attack against Neural Networks
• CopyCAT: Taking Control of Neural Policies with Constant Attacks
• Adaptive Discretization in Online Reinforcement Learning
• Explaining Bayesian Neural Networks
• Erdos Goes Neural: an Unsupervised Learning Framework for Combinatorial Optimization on Graphs
• Modern Deep Reinforcement Learning Algorithms
• Tunneling Neural Perception and Logic Reasoning through Abductive Learning
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Reframing Jet Physics with New Computational Methods
• Monte Carlo Q-learning for General Game Playing
• Intersymbolic AI: Interlinking Symbolic AI and Subsymbolic AI
• Learning to play the Chess Variant Crazyhouse above World Champion Level with Deep Neural Networks and Human Data
• Reinforcement Learning with A* and a Deep Heuristic
• Learning to run a Power Network Challenge: a Retrospective Analysis
• Event Extraction with Generative Adversarial Imitation Learning
• On Multi-Agent Learning in Team Sports Games
• Reinforcement Learning-Driven Test Generation for Android GUI Applications using Formal Specifications
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Synthesized Policies for Transfer and Adaptation across Tasks and Environments
• Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithms
• Student-Initiated Action Advising via Advice Novelty
• Solving Rubik's Cube via Quantum Mechanics and Deep Reinforcement Learning
• Winning Isn't Everything: Enhancing Game Development with Intelligent Agents
• AlphaZero-Inspired Game Learning: Faster Training by Using MCTS Only at Test Time
• Compositional ADAM: An Adaptive Compositional Solver
• Affordance-based Reinforcement Learning for Urban Driving
• Robust Reinforcement Learning in POMDPs with Incomplete and Noisy Observations
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Adaptive Discretization for Model-Based Reinforcement Learning
• Metric Flows with Neural Networks
• Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• Automated Theorem Proving in Intuitionistic Propositional Logic by Deep Reinforcement Learning
• Decentralized Q-Learning in Zero-sum Markov Games
• Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning
• On the Complexity of Reconnaissance Blind Chess
• Hyper-Parameter Sweep on AlphaZero General
• Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning
• Machine Learning Challenges and Opportunities in the African Agricultural Sector -- A General Perspective
• Learning to Communicate in Multi-Agent Reinforcement Learning : A Review
• Stochastic reserving with a stacked model based on a hybridized Artificial Neural Network
• Scaling Imitation Learning in Minecraft
• Learning hierarchical behavior and motion planning for autonomous driving
• HUGO -- Highlighting Unseen Grid Options: Combining Deep Reinforcement Learning with a Heuristic Target Topology Approach
• Finding the best design parameters for optical nanostructures using reinforcement learning
• Improving Movement Predictions of Traffic Actors in Bird's-Eye View Models using GANs and Differentiable Trajectory Rasterization
• An Adaptive and Near Parameter-free Evolutionary Computation Approach Towards True Automation in AutoML
• Experimental design for MRI by greedy policy search
• Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applications
• Deep Quality-Value (DQV) Learning
• Information Theoretic Model Predictive Q-Learning
• Continuous Control with Deep Reinforcement Learning for Autonomous Vessels
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• A Versatile Multi-Robot Monte Carlo Tree Search Planner for On-Line Coverage Path Planning
• Demystifying the MLPerf Benchmark Suite
• Transfer Learning versus Multi-agent Learning regarding Distributed Decision-Making in Highway Traffic
• Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• Exponential improvements for quantum-accessible reinforcement learning
• Corrigibility with Utility Preservation
• Neural Fictitious Self-Play on ELF Mini-RTS
• The 30-Year Cycle In The AI Debate
• Explainable AI: Deep Reinforcement Learning Agents for Residential Demand Side Cost Savings in Smart Grids
• Solving Hard AI Planning Instances Using Curriculum-Driven Deep Reinforcement Learning
• ${\alpha }^{\alpha }$-Rank: Practically Scaling $\alpha$-Rank through Stochastic Optimisation
• Neural Complexity Measures
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits
• Interferobot: aligning an optical interferometer by a reinforcement learning agent
• Learning Agile Locomotion via Adversarial Training
• State-of-Charge Estimation of a Li-Ion Battery using Deep Forward Neural Networks
• Convergence of Multi-Agent Learning with a Finite Step Size in General-Sum Games
• HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents
• StarCraft II Build Order Optimization using Deep Reinforcement Learning and Monte-Carlo Tree Search
• Extractive Summary as Discrete Latent Variables
• RL STaR Platform: Reinforcement Learning for Simulation based Training of Robots
• Reinforcement Learning Enhanced Quantum-inspired Algorithm for Combinatorial Optimization
• Train on Small, Play the Large: Scaling Up Board Games with AlphaZero and GNN
• Self-play Learning Strategies for Resource Assignment in Open-RAN Networks
• The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems
• Skin disease diagnosis with deep learning: a review
• Graceful Degradation and Related Fields
• LiveChess2FEN: a Framework for Classifying Chess Pieces based on CNNs
• Runtime Verification of Learning Properties for Reinforcement Learning Algorithms
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
• On Learning to Prove
• On Reinforcement Learning for Turn-based Zero-sum Markov Games
• Competitive Experience Replay
• Policy Smoothing for Provably Robust Reinforcement Learning
• Towards Mixed Optimization for Reinforcement Learning with Program Synthesis
• Blending MPC & Value Function Approximation for Efficient Reinforcement Learning
• Planning from Pixels using Inverse Dynamics Models
• Towards Understanding Chinese Checkers with Heuristics, Monte Carlo Tree Search, and Deep Reinforcement Learning
• ML + FV = $♡$? A Survey on the Application of Machine Learning to Formal Verification
• Continuous Control for Searching and Planning with a Learned Model
• Playing Chess with Limited Look Ahead
• From Gameplay to Symbolic Reasoning: Learning SAT Solver Heuristics in the Style of Alpha(Go) Zero
• Multi-Agent Deep Reinforcement Learning with Adaptive Policies
• Sample Efficient Ensemble Learning with Catalyst.RL
• Solving optimal stopping problems with Deep Q-Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Generalised agent for solving higher board states of tic tac toe using Reinforcement Learning
• Hierarchical clustering in particle physics through reinforcement learning
• Assessing the Potential of Classical Q-learning in General Game Playing
• AlphaGomoku: An AlphaGo-based Gomoku Artificial Intelligence using Curriculum Learning
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Reducing catastrophic forgetting when evolving neural networks
• Generalization to Novel Objects using Prior Relational Knowledge
• Policy Optimization Reinforcement Learning with Entropy Regularization
• Typed Graph Networks
• AlphaSeq: Sequence Discovery with Deep Reinforcement Learning
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS
• Learning Compositional Neural Programs for Continuous Control
• Learning to Play Two-Player Perfect-Information Games without Knowledge
• Feudal Steering: Hierarchical Learning for Steering Angle Prediction
• Ignorance-Aware Approaches and Algorithms for Prototype Selection in Machine Learning
• The Matthew Effect in Computation Contests: High Difficulty May Lead to 51% Dominance
• Utility Ghost: Gamified redistricting with partisan symmetry
• Multi-Issue Bargaining With Deep Reinforcement Learning
• Finite Group Equivariant Neural Networks for Games
• Time Adaptive Reinforcement Learning
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Mastering Terra Mystica: Applying Self-Play to Multi-agent Cooperative Board Games
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models
• Scaffolding Reflection in Reinforcement Learning Framework for Confinement Escape Problem
• Augmented Shortcuts for Vision Transformers
• Learning proofs for the classification of nilpotent semigroups
• Mack-Net model: Blending Mack's model with Recurrent Neural Networks
• BridgeHand2Vec Bridge Hand Representation
• The Fair Game: Auditing & Debiasing AI Algorithms Over Time
• Mixture of Public and Private Distributions in Imperfect Information Games
• Physics-Inspired Interpretability Of Machine Learning Models
• Reinforcement Learning on Human Decision Models for Uniquely Collaborative AI Teammates
• Search in Imperfect Information Games
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction
• Hierarchical RNNs-Based Transformers MADDPG for Mixed Cooperative-Competitive Environments
• REGRAD: A Large-Scale Relational Grasp Dataset for Safe and Object-Specific Robotic Grasping in Clutter
• Convex Regularization in Monte-Carlo Tree Search
• POLY-HOOT: Monte-Carlo Planning in Continuous Space MDPs with Non-Asymptotic Analysis
• Optimal Immunization Policy Using Dynamic Programming
• Proposal Towards a Personalized Knowledge-powered Self-play Based Ensemble Dialog System
• Reinforcement Learning and Video Games
• Reproducibility via Crowdsourced Reverse Engineering: A Neural Network Case Study With DeepMind's Alpha Zero
• Exploring the Performance of Deep Residual Networks in Crazyhouse Chess
• SentiMATE: Learning to play Chess through Natural Language Processing
• Improving width-based planning with compact policies
• Deep Pepper: Expert Iteration based Chess agent in the Reinforcement Learning Setting
• Single-step Options for Adversary Driving
• Robby is Not a Robber (anymore): On the Use of Institutions for Learning Normative Behavior
• DefogGAN: Predicting Hidden Information in the StarCraft Fog of War with Generative Adversarial Nets
• Comparison Training for Computer Chinese Chess
• Visibility Optimization for Surveillance-Evasion Games
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning
• DriverGym: Democratising Reinforcement Learning for Autonomous Driving
• Quadratic speedup of global search using a biased crossover of two good solutions
• Perfect Information Monte Carlo with Postponing Reasoning
• Enhancing Reinforcement Learning Through Guided Search
• Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi
• Learning to Play Foosball: System and Baselines
• Monte Carlo Tree Search with Scalable Simulation Periods for Continuously Running Tasks
• Room Clearance with Feudal Hierarchical Reinforcement Learning
• Deep Reinforcement Learning with Adjustments
• Learning Cooperation and Online Planning Through Simulation and Graph Convolutional Network
• Deep Synoptic Monte Carlo Planning in Reconnaissance Blind Chess
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• Solving QSAT problems with neural MCTS
• Growing Action Spaces
• Detection of chaotic behavior in time series
• Analyzing the Hidden Activations of Deep Policy Networks: Why Representation Matters
• Neural Networks and Denotation
• Near-optimal Bayesian Solution For Unknown Discrete Markov Decision Process
• On the Power of Refined Skat Selection
• Searching with Opponent-Awareness
• Pyfectious: An individual-level simulator to discover optimal containment polices for epidemic diseases
• Evolution of Q Values for Deep Q Learning in Stable Baselines
• Marginal Utility for Planning in Continuous or Large Discrete Action Spaces
• Safety Aware Reinforcement Learning (SARL)
• Automated Chess Commentator Powered by Neural Chess Engine
• Hierarchical Reinforcement Learning: Approximating Optimal Discounted TSP Using Local Policies
• Autonomous Industrial Management via Reinforcement Learning: Self-Learning Agents for Decision-Making -- A Review
• Self-Play Learning Without a Reward Metric
• Deep RL Agent for a Real-Time Action Strategy Game
• Neural-Network Guided Expression Transformation
• Network Horizon Dynamics I: Qualitative Aspects
• Tiyuntsong: A Self-Play Reinforcement Learning Approach for ABR Video Streaming
• Learning Position Evaluation Functions Used in Monte Carlo Softmax Search