GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model

Abstract

Safe Reinforcement Learning (SRL) aims to realize a safe learning process for Deep Reinforcement Learning (DRL) algorithms by incorporating safety constraints. However, the efficacy of SRL approaches often relies on accurate function approximations, which are notably challenging to achieve in the early learning stages due to data insufficiency. To address this issue, we introduce in this work a novel Generalizable Safety enhancer (GenSafe) that is able to overcome the challenge of data insufficiency and enhance the performance of SRL approaches. Leveraging model order reduction techniques, we first propose an innovative method to construct a Reduced Order Markov Decision Process (ROMDP) as a low-dimensional approximator of the original safety constraints. Then, by solving the reformulated ROMDP-based constraints, GenSafe refines the actions of the agent to increase the possibility of constraint satisfaction. Essentially, GenSafe acts as an additional safety layer for SRL algorithms. We evaluate GenSafe on multiple SRL approaches and benchmark problems. The results demonstrate its capability to improve safety performance, especially in the early learning phases, while maintaining satisfactory task performance. Our proposed GenSafe not only offers a novel measure to augment existing SRL methods but also shows broad compatibility with various SRL algorithms, making it applicable to a wide range of systems and SRL problems.

References in corpus (22)

• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Exploration in Deep Reinforcement Learning: A Survey
• Safe Reinforcement Learning Using Robust MPC
• Safe Exploration in Continuous Action Spaces
• Reward Constrained Policy Optimization
• Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• A Review of Safe Reinforcement Learning: Methods, Theory and Applications
• Projection-Based Constrained Policy Optimization
• Safe Reinforcement Learning in Constrained Markov Decision Processes
• First Order Constrained Optimization in Policy Space
• Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems
• Constrained Update Projection Approach to Safe Policy Optimization
• Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark
• Constrained Policy Optimization via Bayesian World Models
• Representation Learning for Online and Offline RL in Low-rank MDPs
• Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents