On the role of planning in model-based deep reinforcement learning

Abstract

Model-based planning is often thought to be necessary for deep, careful reasoning and generalization in artificial agents. While recent successes of model-based reinforcement learning (MBRL) with deep function approximation have strengthened this hypothesis, the resulting diversity of model-based methods has also made it difficult to track which components drive success and why. In this paper, we seek to disentangle the contributions of recent methods by focusing on three questions: (1) How does planning benefit MBRL agents? (2) Within planning, what choices drive performance? (3) To what extent does planning improve generalization? To answer these questions, we study the performance of MuZero (Schrittwieser et al., 2019), a state-of-the-art MBRL algorithm with strong connections and overlapping components with many other MBRL algorithms. We perform a number of interventions and ablations of MuZero across a wide range of environments, including control tasks, Atari, and 9x9 Go. Our results suggest the following: (1) Planning is most useful in the learning process, both for policy updates and for providing a more useful data distribution. (2) Using shallow trees with simple Monte-Carlo rollouts is as performant as more complex methods, except in the most difficult reasoning tasks. (3) Planning alone is insufficient to drive strong generalization. These results indicate where and how to utilize planning in reinforcement learning settings, and highlight a number of open questions for future MBRL research.

arXiv commentPublished at ICLR 2021

References in corpus (18)

• Trust Region Policy Optimization
• DeepMind Control Suite
• Recurrent World Models Facilitate Policy Evolution
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Benchmarking Model-Based Reinforcement Learning
• On-line Policy Improvement using Monte-Carlo Search
• When to use parametric models in reinforcement learning?
• Acme: A Research Framework for Distributed Reinforcement Learning
• Approximate Policy Iteration Schemes: A Comparison
• Monte-Carlo Tree Search as Regularized Policy Optimization
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
• The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces
• Mirror Descent Policy Optimization
• Reinforcement Learning by Value Gradients
• Dual Policy Iteration
• Policy Gradient Search: Online Planning and Expert Iteration without Search Trees
• Surprising Negative Results for Generative Adversarial Tree Search
• Local Search for Policy Iteration in Continuous Control

Cited by in corpus (10)

• On the role of theory and modeling in neuroscience
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• On the model-based stochastic value gradient for continuous reinforcement learning
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Muesli: Combining Improvements in Policy Optimization
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Transfer Learning and Curriculum Learning in Sokoban
• Planning and Learning Using Adaptive Entropy Tree Search
• Subgoal Search For Complex Reasoning Tasks
• Neural Algorithmic Reasoners are Implicit Planners