Efficient exploration with Double Uncertain Value Networks

Abstract

This paper studies directed exploration for reinforcement learning agents by tracking uncertainty about the value of each available action. We identify two sources of uncertainty that are relevant for exploration. The first originates from limited data (parametric uncertainty), while the second originates from the distribution of the returns (return uncertainty). We identify methods to learn these distributions with deep neural networks, where we estimate parametric uncertainty with Bayesian drop-out, while return uncertainty is propagated through the Bellman equation as a Gaussian distribution. Then, we identify that both can be jointly estimated in one network, which we call the Double Uncertain Value Network. The policy is directly derived from the learned distributions based on Thompson sampling. Experimental results show that both types of uncertainty may vastly improve learning in domains with a strong exploration challenge.

Deep Reinforcement Learning Symposium @ Conference on Neural Information Processing Systems (NIPS) 2017

References in corpus (11)

• Weight Uncertainty in Neural Networks
• VIME: Variational Information Maximizing Exploration
• Deep Exploration via Bootstrapped DQN
• Parameter Space Noise for Exploration
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Model-Based Bayesian Exploration
• Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search
• Generalization and Exploration via Randomized Value Functions
• The Uncertainty Bellman Equation and Exploration
• Parametric Return Density Estimation for Reinforcement Learning
• Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning

Cited by in corpus (7)

• Implicit Quantile Networks for Distributional Reinforcement Learning
• Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
• A0C: Alpha Zero in Continuous Action Space
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• The Potential of the Return Distribution for Exploration in RL
• ADER:Adapting between Exploration and Robustness for Actor-Critic Methods
• Unbiased Deep Reinforcement Learning: A General Training Framework for Existing and Future Algorithms