Policy Gradient for Coherent Risk Measures

Abstract

Several authors have recently developed risk-sensitive policy gradient methods that augment the standard expected cost minimization problem with a measure of variability in cost. These studies have focused on specific risk-measures, such as the variance or conditional value at risk (CVaR). In this work, we extend the policy gradient method to the whole class of coherent risk measures, which is widely accepted in finance and operations research, among other fields. We consider both static and time-consistent dynamic risk measures. For static risk measures, our approach is in the spirit of policy gradient algorithms and combines a standard sampling approach with convex programming. For dynamic risk measures, our approach is actor-critic style and involves explicit approximation of value function. Most importantly, our contribution presents a unified approach to risk-sensitive reinforcement learning that generalizes and extends previous results.

References in corpus (3)

• Algorithms for CVaR Optimization in MDPs
• An Approximate Solution Method for Large Risk-Averse Markov Decision Processes
• Tight Approximations of Dynamic Risk Measures

Cited by in corpus (17)

• Challenges of Real-World Reinforcement Learning
• Inverse Reward Design
• An empirical investigation of the challenges of real-world reinforcement learning
• Distributionally Robust Reinforcement Learning
• Minimizing Spectral Risk Measures Applied to Markov Decision Processes
• Distributional Robustness and Regularization in Reinforcement Learning
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Off-Policy Risk Assessment in Contextual Bandits
• Cautious Reinforcement Learning via Distributional Risk in the Dual Domain
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
• Risk-Averse Bayes-Adaptive Reinforcement Learning
• On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk
• Cutting Your Losses: Learning Fault-Tolerant Control and Optimal Stopping under Adverse Risk
• A Natural Actor-Critic Algorithm with Downside Risk Constraints
• Entropic Risk Constrained Soft-Robust Policy Optimization
• Soft-Robust Algorithms for Batch Reinforcement Learning
• Deep Reinforcement Learning for Equal Risk Pricing and Hedging under Dynamic Expectile Risk Measures