Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
arXiv:2010.14641
Abstract
Learning complex robot behaviors through interaction requires structured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective. This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine latent world models with value function estimation to predict infinite-horizon returns and recover associated uncertainty via ensembling. The policy is then trained on an upper confidence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance. We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efficiency in comparison to state-of-the-art and other exploration objectives. In sparse and hard to explore environments we achieve an average improvement of over 30%.
References in corpus (10)
- Learning agile and dynamic motor skills for legged robots
- DeepMind Control Suite
- Reinforcement Learning with Augmented Data
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
- Planning to Explore via Self-Supervised World Models
- SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
- Data Efficient Reinforcement Learning for Legged Robots
- Explicit Explore-Exploit Algorithms in Continuous State Spaces
- Ready Policy One: World Building Through Active Learning