Learning to be Safe: Deep RL with a Safety Critic

Abstract

Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself. A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. However, just as learning has enabled progress in large-scale development of AI systems, learning safety specifications may also be necessary to ensure safety in messy open-world environments where manual safety specifications cannot scale. Akin to how humans learn incrementally starting in child-safe environments, we propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors when learning new, modified tasks. We empirically study this form of safety-constrained transfer learning in three challenging domains: simulated navigation, quadruped locomotion, and dexterous in-hand manipulation. In comparison to standard deep RL techniques and prior approaches to safe RL, we find that our method enables the learning of new tasks and in new environments with both substantially fewer safety incidents, such as falling or dropping an object, and faster, more stable learning. This suggests a path forward not only for safer RL systems, but also for more effective RL systems.

arXiv commentIn submission, 16 pages (including appendix)

References in corpus (10)

• Soft Actor-Critic Algorithms and Applications
• Robust Adversarial Reinforcement Learning
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Path integrals and symmetry breaking for optimal control theory
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Algorithms for CVaR Optimization in MDPs
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Worst Cases Policy Gradients
• Distributionally Robust Reinforcement Learning
• Verifying Aircraft Collision Avoidance Neural Networks Through Linear Approximations of Safe Regions

Cited by in corpus (15)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Conservative Safety Critics for Exploration
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• Conservative Policy Construction Using Variational Autoencoders for Logged Data with Missing Values
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Safe Autonomous Racing via Approximate Reachability on Ego-vision
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Safe Driving via Expert Guided Policy Optimization
• Safe Exploration by Solving Early Terminated MDP
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• Learning to Be Cautious
• Auditing Robot Learning for Safety and Compliance during Deployment
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control