Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach

Abstract

In this paper we address the problem of decision making within a Markov decision process (MDP) framework where risk and modeling errors are taken into account. Our approach is to minimize a risk-sensitive conditional-value-at-risk (CVaR) objective, as opposed to a standard risk-neutral expectation. We refer to such problem as CVaR MDP. Our first contribution is to show that a CVaR objective, besides capturing risk sensitivity, has an alternative interpretation as expected cost under worst-case modeling errors, for a given error budget. This result, which is of independent interest, motivates CVaR MDPs as a unifying framework for risk-sensitive and robust decision making. Our second contribution is to present an approximate value-iteration algorithm for CVaR MDPs and analyze its convergence rate. To our knowledge, this is the first solution algorithm for CVaR MDPs that enjoys error guarantees. Finally, we present results from numerical experiments that corroborate our theoretical findings and show the practicality of our approach.

Submitted to NIPS 15

References in corpus (2)

• Algorithms for CVaR Optimization in MDPs
• Lightning Does Not Strike Twice: Robust MDPs with Coupled Uncertainty

Cited by in corpus (32)

• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Artificial Intelligence for Social Good
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Worst Cases Policy Gradients
• Risk-Aware Active Inverse Reinforcement Learning
• A Bayesian Approach to Robust Reinforcement Learning
• CoinDICE: Off-Policy Confidence Interval Estimation
• Minimizing Spectral Risk Measures Applied to Markov Decision Processes
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Robust Reinforcement Learning using Adversarial Populations
• Wasserstein Distributionally Robust Motion Control for Collision Avoidance Using Conditional Value-at-Risk
• STEP: Stochastic Traversability Evaluation and Planning for Risk-Aware Off-road Navigation
• Constrained Reinforcement Learning Has Zero Duality Gap
• Improving Robustness via Risk Averse Distributional Reinforcement Learning
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
• Markov Decision Processes under Ambiguity
• Reward Constrained Interactive Recommendation with Natural Language Feedback
• Safety-Aware Optimal Control of Stochastic Systems Using Conditional Value-at-Risk
• Risk-Averse Planning via CVaR Barrier Functions: Application to Bipedal Robot Locomotion
• On the dynamic representation of some time-inconsistent risk measures in a Brownian filtration
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• A Survey of Learning Criteria Going Beyond the Usual Risk
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
• Robust Batch Policy Learning in Markov Decision Processes
• Quantile Markov Decision Process
• Learning with CVaR-based feedback under potentially heavy tails
• Learning to Be Cautious
• SafeAR: Safe Algorithmic Recourse by Risk-Aware Policies
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Biased Estimates of Advantages over Path Ensembles