Worst Cases Policy Gradients

Abstract

Recent advances in deep reinforcement learning have demonstrated the capability of learning complex control policies from many types of environments. When learning policies for safety-critical applications, it is essential to be sensitive to risks and avoid catastrophic events. Towards this goal, we propose an actor-critic framework that models the uncertainty of the future and simultaneously learns a policy based on that uncertainty model. Specifically, given a distribution of the future return for any state and action, we optimize policies for varying levels of conditional Value-at-Risk. The learned policy can map the same state to different actions depending on the propensity for risk. We demonstrate the effectiveness of our approach in the domain of driving simulations, where we learn maneuvers in two scenarios. Our learned controller can dynamically select actions along a continuous axis, where safe and conservative behaviors are found at one end while riskier behaviors are found at the other. Finally, when testing with very different simulation parameters, our risk-averse policies generalize significantly better compared to other reinforcement learning approaches.

Conference on Robot Learning 2019 (CoRL 2019)

References in corpus (7)

• On a Formal Model of Safe and Scalable Self-driving Cars
• Robust Adversarial Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Sample Efficient Actor-Critic with Experience Replay
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• Algorithms for CVaR Optimization in MDPs
• Deep Reinforcement Learning in Parameterized Action Space

Cited by in corpus (14)

• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Learning to be Safe: Deep RL with a Safety Critic
• TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• GIN: Graph-based Interaction-aware Constraint Policy Optimization for Autonomous Driving
• Lyapunov Barrier Policy Optimization
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Risk-Averse Bayes-Adaptive Reinforcement Learning
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Risk-Aware Continuous Control with Neural Contextual Bandits
• Bayesian Robust Optimization for Imitation Learning
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• Learning to Be Cautious