Long-Run Conditional Value at Risk Reinforcement Learning

Abstract

We propose a reinforcement learning method for minimizing the long-run conditional value at risk (CVaR) in Markov decision processes with finite state and action spaces. The method uses a single sample trajectory to update a quantile estimate, a $Q$-function, and a randomized policy, without requiring knowledge of transition probabilities or cost distributions. Under appropriate conditions, we show that the estimates track their changing targets as the policy is updated and exploration decreases, even when the proportion of updates for some state and action pairs tends to zero. With a strict action gap, the policy converges to a locally optimal deterministic policy if it enters a suitable neighborhood sufficiently late. We also extend the method to an objective combining CVaR and mean cost. Numerical experiments illustrate its performance in machine replacement and energy storage and compare it with existing methods.