Planning and Learning Using Adaptive Entropy Tree Search

Abstract

Recent breakthroughs in Artificial Intelligence have shown that the combination of tree-based planning with deep learning can lead to superior performance. We present Adaptive Entropy Tree Search (ANTS) - a novel algorithm combining planning and learning in the maximum entropy paradigm. Through a comprehensive suite of experiments on the Atari benchmark we show that ANTS significantly outperforms PUCT, the planning component of the state-of-the-art AlphaZero system. ANTS builds upon recent work on maximum entropy planning methods - which however, as we show, fail in combination with learning. ANTS resolves this issue to reach state-of-the-art performance. We further find that ANTS exhibits superior robustness to different hyperparameter choices, compared to the previous algorithms. We believe that the high performance and robustness of ANTS can bring tree search planning one step closer to wide practical adoption.

References in corpus (13)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Recurrent World Models Facilitate Policy Evolution
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Agent57: Outperforming the Atari Human Benchmark
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Mastering Atari Games with Limited Data
• Monte-Carlo Tree Search as Regularized Policy Optimization
• If MaxEnt RL is the Answer, What is the Question?
• Combining Q-Learning and Search with Amortized Value Estimates
• On the role of planning in model-based deep reinforcement learning
• Uncertainty-sensitive Learning and Planning with Ensembles