Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning

Abstract

Reach-avoid optimal control problems, in which the system must reach certain goal conditions while staying clear of unacceptable failure modes, are central to safety and liveness assurance for autonomous robotic systems, but their exact solutions are intractable for complex dynamics and environments. Recent successes in reinforcement learning methods to approximately solve optimal control problems with performance objectives make their application to certification problems attractive; however, the Lagrange-type objective used in reinforcement learning is not suitable to encode temporal logic requirements. Recent work has shown promise in extending the reinforcement learning machinery to safety-type problems, whose objective is not a sum, but a minimum (or maximum) over time. In this work, we generalize the reinforcement learning formulation to handle all optimal control problems in the reach-avoid category. We derive a time-discounted reach-avoid Bellman backup with contraction mapping properties and prove that the resulting reach-avoid Q-learning algorithm converges under analogous conditions to the traditional Lagrange-type problem, yielding an arbitrarily tight conservative approximation to the reach-avoid set. We further demonstrate the use of this formulation with deep reinforcement learning methods, retaining zero-violation guarantees by treating the approximate solutions as untrusted oracles in a model-predictive supervisory control framework. We evaluate our proposed framework on a range of nonlinear systems, validating the results against analytic and numerical solutions, and through Monte Carlo simulation in previously intractable problems. Our results open the door to a range of learning-based methods for safe-and-live autonomous behavior, with applications across robotics and automation. See https://github.com/SafeRoboticsLab/safety_rl for code and supplementary material.

arXiv commentAccepted in Robotics: Science and Systems (RSS), 2021

References in corpus (5)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• OpenAI Gym
• Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamics
• Recursive Regression with Neural Networks: Approximating the HJI PDE Solution

Cited by in corpus (7)

• GCBF+: A Neural Graph Control Barrier Function Framework for Distributed Safe Multi-Agent Control
• Safe Value Functions
• Safety on the Fly: Constructing Robust Safety Filters via Policy Control Barrier Functions at Runtime
• DualGuard MPPI: Safe and Performant Optimal Control by Combining Sampling-Based MPC and Hamilton-Jacobi Reachability
• Provably Correct Training of Neural Network Controllers Using Reachability Analysis
• Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities
• Receding-Constraint Model Predictive Control using a Learned Approximate Control-Invariant Set