Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
arXiv:2010.14641
Abstract
Learning complex robot behaviors through interaction requires structured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective. This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine latent world models with value function estimation to predict infinite-horizon returns and recover associated uncertainty via ensembling. The policy is then trained on an upper confidence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance. We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efficiency in comparison to state-of-the-art and other exploration objectives. In sparse and hard to explore environments we achieve an average improvement of over 30%.
References in corpus (26)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Learning agile and dynamic motor skills for legged robots
- DeepMind Control Suite
- VIME: Variational Information Maximizing Exploration
- Count-Based Exploration with Neural Density Models
- Learning Latent Dynamics for Planning from Pixels
- Distributed Distributional Deterministic Policy Gradients
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Model-Ensemble Trust-Region Policy Optimization
- Reinforcement Learning with Augmented Data
- Deep Exploration via Randomized Value Functions
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Model-Based Active Exploration
- Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
- Planning to Explore via Self-Supervised World Models
- SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
- Data Efficient Reinforcement Learning for Legged Robots
- Explicit Explore-Exploit Algorithms in Continuous State Spaces
- Learning Awareness Models
- Ready Policy One: World Building Through Active Learning