The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces

Abstract

Dyna is a fundamental approach to model-based reinforcement learning (MBRL) that interleaves planning, acting, and learning in an online setting. In the most typical application of Dyna, the dynamics model is used to generate one-step transitions from selected start states from the agent's history, which are used to update the agent's value function or policy as if they were real experiences. In this work, one-step Dyna was applied to several games from the Arcade Learning Environment (ALE). We found that the model-based updates offered surprisingly little benefit over simply performing more updates with the agent's existing experience, even when using a perfect model. We hypothesize that to get the most from planning, the model must be used to generate unfamiliar experience. To test this, we experimented with the "shape" of planning in multiple different concrete instantiations of Dyna, performing fewer, longer rollouts, rather than many short rollouts. We found that planning shape has a profound impact on the efficacy of Dyna for both perfect and learned models. In addition to these findings regarding Dyna in general, our results represent, to our knowledge, the first time that a learned dynamics model has been successfully used for planning in the ALE, suggesting that Dyna may be a viable approach to MBRL in the ALE and other high-dimensional problems.

References in corpus (5)

• Auto-Encoding Variational Bayes
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Continuous Deep Q-Learning with Model-based Acceleration
• Agnostic System Identification for Model-Based Reinforcement Learning
• A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Games

Cited by in corpus (12)

• Model-Based Reinforcement Learning for Atari
• MOPO: Model-based Offline Policy Optimization
• When to Trust Your Model: Model-Based Policy Optimization
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• On the role of planning in model-based deep reinforcement learning
• Surprising Negative Results for Generative Adversarial Tree Search
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for Atari
• Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models
• Planning with Expectation Models for Control
• Self-Consistent Models and Values
• Hill Climbing on Value Estimates for Search-control in Dyna