Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

Abstract

Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees. We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward. The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model. The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires \textit{no explicit} uncertainty quantification. Instantiating our framework with simplification gives a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves state-of-the-art performance when only one million or fewer samples are permitted on a range of continuous control benchmark tasks.

arXiv commentAdded important notes that the conditions of Theorem 3.1 cannot simultaneously hold for most models

References in corpus (22)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Continuous Deep Q-Learning with Model-based Acceleration
• Graph networks as learnable physics engines for inference and control
• Model-Ensemble Trust-Region Policy Optimization
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Simple random search provides a competitive approach to reinforcement learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs
• Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning
• Constrained Policy Optimization
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• On the Sample Complexity of the Linear Quadratic Regulator
• Universal Planning Networks
• Learning model-based planning from scratch
• Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification
• Lipschitz Continuity in Model-based Reinforcement Learning
• Variance Reduction Methods for Sublinear Reinforcement Learning

Cited by in corpus (62)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Off-Policy Deep Reinforcement Learning without Exploration
• MOPO: Model-based Offline Policy Optimization
• Benchmarking Model-Based Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• When to Trust Your Model: Model-Based Policy Optimization
• Exploring Model-based Planning with Policy Networks
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Invariant Causal Prediction for Block MDPs
• Combating the Compounding-Error Problem with a Multi-step Model
• Information Theoretic Regret Bounds for Online Nonlinear Control
• World Model as a Graph: Learning Latent Landmarks for Planning
• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control
• Trust the Model When It Is Confident: Masked Model-based Actor-Critic
• Error Bounds of Imitating Policies and Environments
• On the Sample Complexity of Stability Constrained Imitation Learning
• Model-based Adversarial Meta-Reinforcement Learning
• Gradient-Aware Model-based Policy Search
• Model-based Reinforcement Learning for Service Mesh Fault Resiliency in a Web Application-level
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• Model-Based Opponent Modeling
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Policy-Aware Model Learning for Policy Gradient Methods
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs
• Model Imitation for Model-Based Reinforcement Learning
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
• Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature
• Meta-Model-Based Meta-Policy Optimization
• Forethought and Hindsight in Credit Assignment
• On Effective Scheduling of Model-based Reinforcement Learning
• MobILE: Model-Based Imitation Learning From Observation Alone
• Exploration via Hindsight Goal Generation
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks
• Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
• CostNet: An End-to-End Framework for Goal-Directed Reinforcement Learning
• Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimization
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• Learning Calibratable Policies using Programmatic Style-Consistency
• Robo-PlaNet: Learning to Poke in a Day
• Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control
• Learning Off-Policy with Online Planning
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• A Contraction Approach to Model-based Reinforcement Learning
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning
• Model Embedding Model-Based Reinforcement Learning
• MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning
• On-Policy Model Errors in Reinforcement Learning
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
• Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning
• Generative Temporal Difference Learning for Infinite-Horizon Prediction