Epistemic Risk-Sensitive Reinforcement Learning

Abstract

We develop a framework for interacting with uncertain environments in reinforcement learning (RL) by leveraging preferences in the form of utility functions. We claim that there is value in considering different risk measures during learning. In this framework, the preference for risk can be tuned by variation of the parameter $\beta$ and the resulting behavior can be risk-averse, risk-neutral or risk-taking depending on the parameter choice. We evaluate our framework for learning problems with model uncertainty. We measure and control for \emph{epistemic} risk using dynamic programming (DP) and policy gradient-based algorithms. The risk-averse behavior is then compared with the behavior of the optimal risk-neutral policy in environments with epistemic risk.

arXiv comment8 pages, 2 figures

References in corpus (5)

• VIME: Variational Information Maximizing Exploration
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
• Combining policy gradient and Q-learning
• Algorithms for CVaR Optimization in MDPs
• Bayesian policy gradient and actor-critic algorithms

Cited by in corpus (5)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Bayesian Robust Optimization for Imitation Learning
• Entropic Risk Constrained Soft-Robust Policy Optimization