Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
arXiv:2004.10888
Abstract
We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.
arXiv commentAAAI 2021
References in corpus (18)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Meta-Gradient Reinforcement Learning
- SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
- Policy Gradients with Variance Related Risk Criteria
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
- Finite-Sample Analysis of Proximal Gradient TD Algorithms
- Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Algorithms for CVaR Optimization in MDPs
- GenDICE: Generalized Offline Estimation of Stationary Values
- Off-Policy Policy Gradient with State Distribution Correction
- Consistent On-Line Off-Policy Evaluation
- On the Finite Time Convergence of Cyclic Coordinate Descent Methods
- A Block Coordinate Ascent Algorithm for Mean-Variance Optimization