Implicit Generative Modeling for Efficient Exploration
arXiv:1911.08017
Abstract
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration. We design a training algorithm for our generative model based on the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efficiency in exploration.
arXiv comment14 pages, 9 figures, Accepted to ICML 2020
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Searching for Activation Functions
- Noisy Networks for Exploration
- VIME: Variational Information Maximizing Exploration
- Parameter Space Noise for Exploration
- Large-Scale Study of Curiosity-Driven Learning
- Count-Based Exploration with Neural Density Models
- Learning Latent Dynamics for Planning from Pixels
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Exploration by Random Network Distillation
- Variational Intrinsic Control
- Model-Based Active Exploration
- Gradient Estimators for Implicit Models
- Provable Bayesian Inference via Particle Mirror Descent
- Learning to Draw Samples with Amortized Stein Variational Gradient Descent
- HyperGAN: A Generative Model for Diverse, Performant Neural Networks
- Wasserstein variational gradient descent: From semi-discrete optimal transport to ensemble variational inference