Monte-Carlo Tree Search for Policy Optimization
arXiv:1912.10648
Abstract
Gradient-based methods are often used for policy optimization in deep reinforcement learning, despite being vulnerable to local optima and saddle points. Although gradient-free methods (e.g., genetic algorithms or evolution strategies) help mitigate these issues, poor initialization and local optima are still concerns in highly nonconvex spaces. This paper presents a method for policy optimization based on Monte-Carlo tree search and gradient-free optimization. Our method, called Monte-Carlo tree search for policy optimization (MCTSPO), provides a better exploration-exploitation trade-off through the use of the upper confidence bound heuristic. We demonstrate improved performance on reinforcement learning tasks with deceptive or sparse reward functions compared to popular gradient-based and deep genetic algorithm baselines.
IJCAI 2019
References in corpus (6)
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Rainbow: Combining Improvements in Deep Reinforcement Learning