Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
arXiv:2006.08684
Abstract
Model-based reinforcement learning algorithms with probabilistic dynamical models are amongst the most data-efficient learning methods. This is often attributed to their ability to distinguish between epistemic and aleatoric uncertainty. However, while most algorithms distinguish these two uncertainties for learning the model, they ignore it when optimizing the policy, which leads to greedy and insufficient exploration. At the same time, there are no practical solvers for optimistic exploration algorithms. In this paper, we propose a practical optimistic exploration algorithm (H-UCRL). H-UCRL reparameterizes the set of plausible models and hallucinates control directly on the epistemic uncertainty. By augmenting the input space with the hallucinated inputs, H-UCRL can be solved using standard greedy planners. Furthermore, we analyze H-UCRL and construct a general regret bound for well-calibrated models, which is provably sublinear in the case of Gaussian Process models. Based on this theoretical foundation, we show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms and different probabilistic models. Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions, a setting that is notoriously difficult for existing model-based reinforcement learning algorithms.
References in corpus (37)
- Auto-Encoding Variational Bayes
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Gaussian Processes for Data-Efficient Learning in Robotics and Control
- DeepMind Control Suite
- Model-Based Reinforcement Learning for Atari
- Accurate Uncertainties for Deep Learning Using Calibrated Regression
- Model-Ensemble Trust-Region Policy Optimization
- Model-Based Bayesian Exploration
- (More) Efficient Reinforcement Learning via Posterior Sampling
- Maximum a Posteriori Policy Optimisation
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- When to Trust Your Model: Model-Based Policy Optimization
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Black box variational inference for state space models
- Batched Large-scale Bayesian Optimization in High-dimensional Spaces
- When to use parametric models in reinforcement learning?
- Monte Carlo Gradient Estimation in Machine Learning
- Generalization and Exploration via Randomized Value Functions
- Exploring Model-based Planning with Policy Networks
- Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds
- Ensemble Sampling
- Model-Augmented Actor-Critic: Backpropagating through Paths
- Information Theoretic Regret Bounds for Online Nonlinear Control
- Lower Bounds on Regret for Noisy Gaussian Process Bandit Optimization
- Calibrated Model-Based Deep Reinforcement Learning
- Uniform Error Bounds for Gaussian Process Regression with Application to Safe Control
- On Simulation and Trajectory Prediction with Gaussian Process Dynamics
- Model-based Lookahead Reinforcement Learning
- Structured Variational Inference in Unstable Gaussian Process State Space Models