Randomized Prior Functions for Deep Reinforcement Learning

Abstract

Dealing with uncertainty is essential for efficient reinforcement learning. There is a growing literature on uncertainty estimation for deep learning from fixed datasets, but many of the most popular approaches are poorly-suited to sequential decision problems. Other methods, such as bootstrap sampling, have no mechanism for uncertainty that does not come from the observed data. We highlight why this can be a crucial shortcoming and propose a simple remedy through addition of a randomized untrainable `prior' network to each ensemble member. We prove that this approach is efficient with linear representations, provide simple illustrations of its efficacy with nonlinear representations and show that this approach scales to large-scale problems far better than previous attempts.

References in corpus (2)

• Distributed Prioritized Experience Replay
• Why is Posterior Sampling Better than Optimism for Reinforcement Learning?

Cited by in corpus (28)

• Exploration in Deep Reinforcement Learning: A Survey
• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• Exploration by Random Network Distillation
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• MOReL : Model-Based Offline Reinforcement Learning
• Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• COMBO: Conservative Offline Model-Based Policy Optimization
• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Materials property prediction using symmetry-labeled graphs as atomic-position independent descriptors
• Scalable Coordinated Exploration in Concurrent Reinforcement Learning
• Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Towards Causality-Aware Inferring: A Sequential Discriminative Approach for Medical Diagnosis
• Automating Reinforcement Learning with Example-based Resets
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning
• Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
• The Potential of the Return Distribution for Exploration in RL
• Uncertainty-sensitive Learning and Planning with Ensembles
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Search
• Bayesian Neural Network Ensembles
• MobILE: Model-Based Imitation Learning From Observation Alone
• Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise
• Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Reward
• Exploring Variational Deep Q Networks