Monte-Carlo Tree Search as Regularized Policy Optimization

Abstract

The combination of Monte-Carlo tree search (MCTS) with deep reinforcement learning has led to significant advances in artificial intelligence. However, AlphaZero, the current state-of-the-art MCTS algorithm, still relies on handcrafted heuristics that are only partially understood. In this paper, we show that AlphaZero's search heuristics, along with other common ones such as UCT, are an approximation to the solution of a specific regularized policy optimization problem. With this insight, we propose a variant of AlphaZero which uses the exact solution to this policy optimization problem, and show experimentally that it reliably outperforms the original algorithm in multiple domains.

Accepted to International Conference on Machine Learning (ICML), 2020

Cited by in corpus (16)

• On the role of planning in model-based deep reinforcement learning
• Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree search
• Learning and Planning in Complex Action Spaces
• Scaling Scaling Laws with Board Games
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Scalable Online Planning via Reinforcement Learning Fine-Tuning
• Procedural Generalization by Planning with Self-Supervised World Models
• Learning compositional programs with arguments and sampling
• Autonomous Port Navigation With Ranging Sensors Using Model-Based Reinforcement Learning
• Dual Monte Carlo Tree Search
• Convex Regularization in Monte-Carlo Tree Search
• Efficient Competitive Self-Play Policy Optimization
• Planning and Learning Using Adaptive Entropy Tree Search
• Towards an Understanding of Default Policies in Multitask Policy Optimization
• ScrofaZero: Mastering Trick-taking Poker Game Gongzhu by Deep Reinforcement Learning
• Combining Off and On-Policy Training in Model-Based Reinforcement Learning