Parametric Return Density Estimation for Reinforcement Learning
arXiv:1203.3497
Abstract
Most conventional Reinforcement Learning (RL) algorithms aim to optimize decision-making rules in terms of the expected returns. However, especially for risk management purposes, other risk-sensitive criteria such as the value-at-risk or the expected shortfall are sometimes preferred in real applications. Here, we describe a parametric method for estimating density of the returns, which allows us to handle various criteria in a unified manner. We first extend the Bellman equation for the conditional expected return to cover a conditional probability density of the returns. Then we derive an extension of the TD-learning algorithm for estimating the return densities in an unknown environment. As test instances, several parametric density estimation algorithms are presented for the Gaussian, Laplace, and skewed Laplace distributions. We show that these algorithms lead to risk-sensitive as well as robust RL paradigms through numerical experiments.
Appears in Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence (UAI2010)
References in corpus (1)
Cited by in corpus (12)
- An Analysis of Categorical Distributional Reinforcement Learning
- Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward Uncertainty
- Efficient exploration with Double Uncertain Value Networks
- The Potential of the Return Distribution for Exploration in RL
- Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks
- Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
- An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
- Universal Off-Policy Evaluation
- Practical Risk Measures in Reinforcement Learning
- Distributional Reinforcement Learning with Ensembles
- A Study of Policy Gradient on a Class of Exactly Solvable Models
- Comparing discounted and average-cost Markov Decision Processes: a statistical significance perspective