A0C: Alpha Zero in Continuous Action Space

Abstract

A core novelty of Alpha Zero is the interleaving of tree search and deep learning, which has proven very successful in board games like Chess, Shogi and Go. These games have a discrete action space. However, many real-world reinforcement learning domains have continuous action spaces, for example in robotic control, navigation and self-driving cars. This paper presents the necessary theoretical extensions of Alpha Zero to deal with continuous action space. We also provide some preliminary experiments on the Pendulum swing-up task, empirically showing the feasibility of our approach. Thereby, this work provides a first step towards the application of iterated search and learning in domains with a continuous action space.

References in corpus (5)

• TensorFlow: A system for large-scale machine learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Efficient exploration with Double Uncertain Value Networks
• Monte Carlo Tree Search for Asymmetric Trees

Cited by in corpus (8)

• Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimization
• Local Search for Policy Iteration in Continuous Control
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Continuous Control for Searching and Planning with a Learned Model
• Parallelization of Monte Carlo Tree Search in Continuous Domains
• Dream and Search to Control: Latent Space Planning for Continuous Control
• Combining Off and On-Policy Training in Model-Based Reinforcement Learning
• AlphaZero Based Post-Storm Repair Crew Dispatch for Distribution Grid Restoration