Algorithms for CVaR Optimization in MDPs

Abstract

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem.

Submitted to NIPS 14

References in corpus (1)

• An Approximate Solution Method for Large Risk-Averse Markov Decision Processes

Cited by in corpus (40)

• Implicit Quantile Networks for Distributional Reinforcement Learning
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• Risk-Aware Energy Scheduling for Edge Computing with Microgrid: A Multi-Agent Deep Reinforcement Learning Approach
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• Measuring the Reliability of Reinforcement Learning Algorithms
• Policy Gradient for Coherent Risk Measures
• Worst Cases Policy Gradients
• Learning to be Safe: Deep RL with a Safety Critic
• Minimizing Spectral Risk Measures Applied to Markov Decision Processes
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Risk-Averse Offline Reinforcement Learning
• Learning Bounds for Risk-sensitive Learning
• PAC-Bayesian Bound for the Conditional Value at Risk
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
• Lyapunov Barrier Policy Optimization
• Improving Robustness via Risk Averse Distributional Reinforcement Learning
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• Epistemic Risk-Sensitive Reinforcement Learning
• Risk-Averse Bayes-Adaptive Reinforcement Learning
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk
• Risk-Averse Planning via CVaR Barrier Functions: Application to Bipedal Robot Locomotion
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Risk-Averse Learning by Temporal Difference Methods
• Quantile Markov Decision Process
• Bias-Corrected Peaks-Over-Threshold Estimation of the CVaR
• Central-limit approach to risk-aware Markov decision processes
• Lyapunov Robust Constrained-MDPs: Soft-Constrained Robustly Stable Policy Optimization under Model Uncertainty
• Learning Robust Options by Conditional Value at Risk Optimization
• Automatic Risk Adaptation in Distributional Reinforcement Learning
• Stochastically Dominant Distributional Reinforcement Learning
• Safe Distributional Reinforcement Learning
• Mean-Variance Efficient Reinforcement Learning with Applications to Dynamic Financial Investment
• Entropic Risk Constrained Soft-Robust Policy Optimization
• Learning Threshold-Type Investment Strategies with Stochastic Gradient Method
• Avoidance Learning Using Observational Reinforcement Learning
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning
• Biased Estimates of Advantages over Path Ensembles
• Soft-Robust Algorithms for Batch Reinforcement Learning