Learning model-based planning from scratch

Abstract

Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does not prescribe how to construct a plan. Here we introduce the "Imagination-based Planner", the first model-based, sequential decision-making agent that can learn to construct, evaluate, and execute plans. Before any action, it can perform a variable number of imagination steps, which involve proposing an imagined action and evaluating it with its model-based imagination. All imagined actions and outcomes are aggregated, iteratively, into a "plan context" which conditions future real and imagined actions. The agent can even decide how to imagine: testing out alternative imagined actions, chaining sequences of actions together, or building a more complex "imagination tree" by navigating flexibly among the previously imagined states using a learned policy. And our agent can learn to plan economically, jointly optimizing for external rewards and computational costs associated with using its imagination. We show that our architecture can learn to solve a challenging continuous control problem, and also learn elaborate planning strategies in a discrete maze-solving task. Our work opens a new direction toward learning the components of a model-based planning system and how to use them.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Adaptive Computation Time for Recurrent Neural Networks
• Conditional Computation in Neural Networks for faster models
• Deep Learning of Representations: Looking Forward
• Learning Visual Predictive Models of Physics for Playing Billiards
• Strategic Attentive Writer for Learning Macro-Actions
• Reasoning About Beliefs and Actions Under Computational Resource Constraints
• Metacontrol for Adaptive Imagination-Based Optimization

Cited by in corpus (36)

• A Brief Survey of Deep Reinforcement Learning
• An Introduction to Deep Reinforcement Learning
• Deep Reinforcement Learning for Cyber Security
• Graph networks as learnable physics engines for inference and control
• Differentiable MPC for End-to-end Planning and Control
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• Relational Deep Reinforcement Learning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Learning Compositional Koopman Operators for Model-Based Control
• How higher goals are constructed and collapse under stress: a hierarchical Bayesian control systems perspective
• Learning to Search with MCTSnets
• Model-Based Planning with Discrete and Continuous Actions
• Combining Q-Learning and Search with Amortized Value Estimates
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
• Muesli: Combining Improvements in Policy Optimization
• Propagation Networks for Model-Based Control Under Partial Observation
• A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making
• Computing a human-like reaction time metric from stable recurrent vision models
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Graph neural induction of value iteration
• Uncertainty-sensitive Learning and Planning with Ensembles
• Generic Itemset Mining Based on Reinforcement Learning
• Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization
• Learning Manipulation under Physics Constraints with Visual Perception
• AutoEG: Automated Experience Grafting for Off-Policy Deep Reinforcement Learning
• Intelligent Trainer for Model-Based Reinforcement Learning
• Structured agents for physical construction
• Policy Gradients Incorporating the Future
• NeoNav: Improving the Generalization of Visual Navigation via Generating Next Expected Observations
• Amanuensis: The Programmer's Apprentice
• Adding Intuitive Physics to Neural-Symbolic Capsules Using Interaction Networks
• Planning in Dynamic Environments with Conditional Autoregressive Models
• Biological Blueprints for Next Generation AI Systems