Estimating Risk and Uncertainty in Deep Reinforcement Learning

Abstract

Reinforcement learning agents are faced with two types of uncertainty. Epistemic uncertainty stems from limited data and is useful for exploration, whereas aleatoric uncertainty arises from stochastic environments and must be accounted for in risk-sensitive applications. We highlight the challenges involved in simultaneously estimating both of them, and propose a framework for disentangling and estimating these uncertainties on learned Q-values. We derive unbiased estimators of these uncertainties and introduce an uncertainty-aware DQN algorithm, which we show exhibits safe learning behavior and outperforms other DQN variants on the MinAtar testbed.

arXiv commentWork presented at the ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning

References in corpus (14)

• Weight Uncertainty in Neural Networks
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• AI Safety Gridworlds
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
• BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Noise Contrastive Priors for Functional Uncertainty
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• Information-Directed Exploration for Deep Reinforcement Learning
• Mitigating Planner Overfitting in Model-Based Reinforcement Learning

Cited by in corpus (16)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
• Risk-Averse Bayes-Adaptive Reinforcement Learning
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy
• Should artificial agents ask for help in human-robot collaborative problem-solving?
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition
• A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty
• Learning to Be Cautious
• Learning to Stop: Dynamic Simulation Monte-Carlo Tree Search
• Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios
• Application of Deep Q-Network in Portfolio Management