A Lyapunov-based Approach to Safe Reinforcement Learning

Abstract

In many real-world reinforcement learning (RL) problems, besides optimizing the main objective function, an agent must concurrently avoid violating a number of constraints. In particular, besides optimizing performance it is crucial to guarantee the safety of an agent during training as well as deployment (e.g. a robot should avoid taking actions - exploratory or not - which irrevocably harm its hardware). To incorporate safety in RL, we derive algorithms under the framework of constrained Markov decision problems (CMDPs), an extension of the standard Markov decision problems (MDPs) augmented with constraints on expected cumulative costs. Our approach hinges on a novel \emph{Lyapunov} method. We define and present a method for constructing Lyapunov functions, which provide an effective way to guarantee the global safety of a behavior policy during training via a set of local, linear constraints. Leveraging these theoretical underpinnings, we show how to use the Lyapunov approach to systematically transform dynamic programming (DP) and RL algorithms into their safe counterparts. To illustrate their effectiveness, we evaluate these algorithms in several CMDP planning and decision-making tasks on a safety benchmark domain. Our results show that our proposed method significantly outperforms existing baselines in balancing constraint satisfaction and performance.

References in corpus (6)

• Safe Exploration in Continuous Action Spaces
• Policy Gradients with Variance Related Risk Criteria
• AI Safety Gridworlds
• Constrained Policy Optimization
• A unified view of entropy-regularized Markov decision processes
• Multi-Objective Deep Reinforcement Learning

Cited by in corpus (23)

• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• Constrained Upper Confidence Reinforcement Learning
• Provably Efficient Q-Learning with Low Switching Cost
• Explanation-Aware Experience Replay in Rule-Dense Environments
• Safe Reinforcement Learning via Probabilistic Shields
• Multi-Agent Constrained Policy Optimisation
• Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation
• Conservative Agency via Attainable Utility Preservation
• Verification of Neural Network Control Policy Under Persistent Adversarial Perturbation
• Actor-Critic Reinforcement Learning for Control with Stability Guarantee
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Avoiding Side Effects in Complex Environments
• Explicit Explore, Exploit, or Escape ($E^4$): near-optimal safety-constrained reinforcement learning in polynomial time
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes
• Lyapunov-Based Reinforcement Learning for Decentralized Multi-Agent Control
• Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning
• Lyapunov-based uncertainty-aware safe reinforcement learning
• Better Safe than Sorry: Evidence Accumulation Allows for Safe Reinforcement Learning
• Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning
• Let's Keep It Safe: Designing User Interfaces that Allow Everyone to Contribute to AI Safety