Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Abstract

We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex simulated control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world.

arXiv commentThe first two authors contributed equally. Accepted at ICLR 2019. Supplementary videos available at: https://sites.google.com/view/polo-mpc

References in corpus (11)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Parameter Space Noise for Exploration
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Unifying Count-Based Exploration and Intrinsic Motivation
• Thinking Fast and Slow with Deep Learning and Tree Search
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Universal Planning Networks
• Agnostic System Identification for Model-Based Reinforcement Learning

Cited by in corpus (35)

• Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control
• Dream to Control: Learning Behaviors by Latent Imagination
• Deep Dynamics Models for Learning Dexterous Manipulation
• Planning to Explore via Self-Supervised World Models
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Model-Based Offline Planning
• Combining Q-Learning and Search with Amortized Value Estimates
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Reset-Free Lifelong Learning with Skill-Space Planning
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• Safe Interactive Model-Based Learning
• Adaptive Online Planning for Continual Lifelong Learning
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Uncertainty-sensitive Learning and Planning with Ensembles
• Information Theoretic Model Predictive Q-Learning
• Regularizing Trajectory Optimization with Denoising Autoencoders
• Multi-Objective Graph Heuristic Search for Terrestrial Robot Design
• Adversarially Regularized Policy Learning Guided by Trajectory Optimization
• Blending MPC & Value Function Approximation for Efficient Reinforcement Learning
• Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians
• ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditions
• Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimization
• Safely Learning Dynamical Systems from Short Trajectories
• Neural Lyapunov Model Predictive Control: Learning Safe Global Controllers from Sub-optimal Examples
• Continuous Control With Ensemble Deep Deterministic Policy Gradients
• Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation
• Dream and Search to Control: Latent Space Planning for Continuous Control
• Learning Off-Policy with Online Planning
• A novel approach to model exploration for value function learning
• Learn a Prior for RHEA for Better Online Planning
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Discriminator Augmented Model-Based Reinforcement Learning
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning