Explicit Explore-Exploit Algorithms in Continuous State Spaces

Abstract

We present a new model-based algorithm for reinforcement learning (RL) which consists of explicit exploration and exploitation phases, and is applicable in large or infinite state spaces. The algorithm maintains a set of dynamics models consistent with current experience and explores by finding policies which induce high disagreement between their state predictions. It then exploits using the refined set of models or experience gathered during exploration. We show that under realizability and optimal planning assumptions, our algorithm provably finds a near-optimal policy with a number of samples that is polynomial in a structural complexity measure which we show to be low in several natural settings. We then give a practical approximation using neural networks and demonstrate its performance and sample efficiency in practice.

Cited by in corpus (9)

• Planning to Explore via Self-Supervised World Models
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
• Adaptive Discretization in Online Reinforcement Learning
• Adaptive Discretization for Model-Based Reinforcement Learning
• Ready Policy One: World Building Through Active Learning
• Meta-Model-Based Meta-Policy Optimization
• Explicit Explore, Exploit, or Escape ($E^4$): near-optimal safety-constrained reinforcement learning in polynomial time
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles