Model-Based Active Exploration

Abstract

Efficient exploration is an unsolved problem in Reinforcement Learning which is usually addressed by reactively rewarding the agent for fortuitously encountering novel situations. This paper introduces an efficient active exploration algorithm, Model-Based Active eXploration (MAX), which uses an ensemble of forward models to plan to observe novel events. This is carried out by optimizing agent behaviour with respect to a measure of novelty derived from the Bayesian perspective of exploration, which is estimated using the disagreement between the futures predicted by the ensemble members. We show empirically that in semi-random discrete environments where directed exploration is critical to make progress, MAX is at least an order of magnitude more efficient than strong baselines. MAX scales to high-dimensional continuous environments where it builds task-agnostic models that can be used for any downstream task.

arXiv commentICML 2019. Code: https://github.com/nnaisense/max

Cited by in corpus (35)

• Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control
• A survey on intrinsic motivation in reinforcement learning
• Planning to Explore via Self-Supervised World Models
• Reinforcement Learning through Active Inference
• On Warm-Starting Neural Network Training
• Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
• Active Multi-Object Exploration and Recognition via Tactile Whiskers
• Curious iLQR: Resolving Uncertainty in Model-based RL
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• BeBold: Exploration Beyond the Boundary of Explored Regions
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Self-Supervised Exploration via Disagreement
• Optimistic Exploration even with a Pessimistic Initialisation
• Exploration and preference satisfaction trade-off in reward-free learning
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Safe Interactive Model-Based Learning
• Ready Policy One: World Building Through Active Learning
• Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation
• Exploratory State Representation Learning
• Information Theoretic Model Predictive Q-Learning
• Generating Automatic Curricula via Self-Supervised Active Domain Randomization
• A Survey of Exploration Methods in Reinforcement Learning
• Implicit Generative Modeling for Efficient Exploration
• Learning more skills through optimistic exploration
• Blending MPC & Value Function Approximation for Efficient Reinforcement Learning
• Online reinforcement learning with sparse rewards through an active inference capsule
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimization
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Bayes-Adaptive Deep Model-Based Policy Optimisation
• Explore and Control with Adversarial Surprise