Policy Gradients with Variance Related Risk Criteria

Abstract

Managing risk in dynamic decision problems is of cardinal importance in many fields such as finance and process control. The most common approach to defining risk is through various variance related criteria such as the Sharpe Ratio or the standard deviation adjusted reward. It is known that optimizing many of the variance related risk criteria is NP-hard. In this paper we devise a framework for local policy gradient style algorithms for reinforcement learning for variance related criteria. Our starting point is a new formula for the variance of the cost-to-go in episodic tasks. Using this formula we develop policy gradient algorithms for criteria that involve both the expected cost and the variance of the cost. We prove the convergence of these algorithms to local minima and demonstrate their applicability in a portfolio planning problem.

Appears in Proceedings of the 29th International Conference on Machine Learning (ICML 2012)

Cited by in corpus (27)

• Challenges of Real-World Reinforcement Learning
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control
• Risk-Constrained Reinforcement Learning with Percentile Risk Criteria
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• TD-Regularized Actor-Critic Methods
• Safe Option-Critic: Learning Safety in the Option-Critic Architecture
• Variance Adjusted Actor Critic Algorithms
• A Demonstration of Issues with Value-Based Multiobjective Reinforcement Learning Under Stochastic State Transitions
• Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach
• Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints
• Constrained Reinforcement Learning Has Zero Duality Gap
• Calibration of Shared Equilibria in General Sum Partially Observable Markov Games
• On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk
• Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods
• Reward Constrained Interactive Recommendation with Natural Language Feedback
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Formal Language Constraints for Markov Decision Processes
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost
• Adaptive Risk Sensitive Model Predictive Control with Stochastic Search
• Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures
• Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets
• Situationally Aware Options
• Curiosity Killed or Incapacitated the Cat and the Asymptotically Optimal Agent
• Mean-Variance Optimization of Discrete Time Discounted Markov Decision Processes