Cautious Reinforcement Learning via Distributional Risk in the Dual Domain

Abstract

We study the estimation of risk-sensitive policies in reinforcement learning problems defined by a Markov Decision Process (MDPs) whose state and action spaces are countably finite. Prior efforts are predominately afflicted by computational challenges associated with the fact that risk-sensitive MDPs are time-inconsistent. To ameliorate this issue, we propose a new definition of risk, which we call caution, as a penalty function added to the dual objective of the linear programming (LP) formulation of reinforcement learning. The caution measures the distributional risk of a policy, which is a function of the policy's long-term state occupancy distribution. To solve this problem in an online model-free manner, we propose a stochastic variant of primal-dual method that uses Kullback-Lieber (KL) divergence as its proximal term. We establish that the number of iterations/samples required to attain approximately optimal solutions of this scheme matches tight dependencies on the cardinality of the state and action spaces, but differs in its dependence on the infinity norm of the gradient of the risk measure. Experiments demonstrate the merits of this approach for improving the reliability of reward accumulation without additional computational burdens.

References in corpus (5)

• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• Policy Gradient for Coherent Risk Measures
• Primal-Dual $\pi$ Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems
• Convergent Policy Optimization for Safe Reinforcement Learning
• A Universal Algorithm for Variational Inequalities Adaptive to Smoothness and Noise

Cited by in corpus (5)

• Variational Policy Gradient Method for Reinforcement Learning with General Utilities
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• MARL with General Utilities via Decentralized Shadow Reward Actor-Critic
• Distributional Reinforcement Learning for Multi-Dimensional Reward Functions
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport