Combining Q-Learning and Search with Amortized Value Estimates

Abstract

We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used in combination with real experience to update the prior. This effectively amortizes the value computation performed by MCTS, resulting in a cooperative relationship between model-free learning and model-based search. SAVE can be implemented on top of any Q-learning agent with access to a model, which we demonstrate by incorporating it into agents that perform challenging physical reasoning tasks and Atari. SAVE consistently achieves higher rewards with fewer training steps, and---in contrast to typical model-based search approaches---yields strong performance with very small search budgets. By combining real experience with information computed during search, SAVE demonstrates that it is possible to improve on both the performance of model-free learning and the computational cost of planning.

Published as a conference paper at ICLR 2020

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning to Plan Chemical Syntheses
• Model-Based Reinforcement Learning for Atari
• Continuous Deep Q-Learning with Model-based Acceleration
• Benchmarking Model-Based Reinforcement Learning
• The Predictron: End-To-End Learning and Planning
• Universal Planning Networks
• Learning model-based planning from scratch
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Learning and Querying Fast Generative Models for Reinforcement Learning
• M-Walk: Learning to Walk over Graphs using Monte Carlo Tree Search
• Policy Gradient Search: Online Planning and Expert Iteration without Search Trees
• Surprising Negative Results for Generative Adversarial Tree Search

Cited by in corpus (4)

• Muesli: Combining Improvements in Policy Optimization
• Plan2Vec: Unsupervised Representation Learning by Latent Plans
• Learning Off-Policy with Online Planning
• Planning and Learning Using Adaptive Entropy Tree Search