Safe Reinforcement Learning via Probabilistic Shields

Abstract

This paper targets the efficient construction of a safety shield for decision making in scenarios that incorporate uncertainty. Markov decision processes (MDPs) are prominent models to capture such planning problems. Reinforcement learning (RL) is a machine learning technique to determine near-optimal policies in MDPs that may be unknown prior to exploring the model. However, during exploration, RL is prone to induce behavior that is undesirable or not allowed in safety- or mission-critical contexts. We introduce the concept of a probabilistic shield that enables decision-making to adhere to safety constraints with high probability. In a separation of concerns, we employ formal verification to efficiently compute the probabilities of critical decisions within a safety-relevant fragment of the MDP. We use these results to realize a shield that is applied to an RL algorithm which then optimizes the actual performance objective. We discuss tradeoffs between sufficient progress in exploration of the environment and ensuring safety. In our experiments, we demonstrate on the arcade game PAC-MAN and on a case study involving service robots that the learning efficiency increases as the learning needs orders of magnitude fewer episodes.

References in corpus (9)

• A Berkeley View of Systems Challenges for AI
• Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigation
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints
• End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks
• Reinforcement Learning with Probabilistic Guarantees for Autonomous Driving
• Shield Synthesis: Runtime Enforcement for Reactive Systems
• Learning Robotic Manipulation through Visual Planning and Acting
• SMarTplan: a Task Planner for Smart Factories

Cited by in corpus (4)

• Control Barriers in Bayesian Learning of System Dynamics
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Formal Language Constraints for Markov Decision Processes
• Learning on Abstract Domains: A New Approach for Verifiable Guarantee in Reinforcement Learning