Risk-Averse Offline Reinforcement Learning

Abstract

Training Reinforcement Learning (RL) agents in high-stakes applications might be too prohibitive due to the risk associated to exploration. Thus, the agent can only use data previously collected by safe policies. While previous work considers optimizing the average performance using offline data, we focus on optimizing a risk-averse criteria, namely the CVaR. In particular, we present the Offline Risk-Averse Actor-Critic (O-RAAC), a model-free RL algorithm that is able to learn risk-averse policies in a fully offline setting. We show that O-RAAC learns policies with higher CVaR than risk-neutral approaches in different robot control tasks. Furthermore, considering risk-averse criteria guarantees distributional robustness of the average performance with respect to particular distribution shifts. We demonstrate empirically that in the presence of natural distribution-shifts, O-RAAC learns policies with good average performance.

References in corpus (7)

• Auto-Encoding Variational Bayes
• Challenges of Real-World Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Algorithms for CVaR Optimization in MDPs
• Risk-Sensitive Generative Adversarial Imitation Learning
• Improving Robustness via Risk Averse Distributional Reinforcement Learning

Cited by in corpus (3)

• Robust Dynamic Bus Control: A Distributional Multi-agent Reinforcement Learning Approach
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Deep Reinforcement Learning for Equal Risk Pricing and Hedging under Dynamic Expectile Risk Measures