A Review of Safe Reinforcement Learning Methods for Modern Power Systems

Abstract

Given the availability of more comprehensive measurement data in modern power systems, reinforcement learning (RL) has gained significant interest in operation and control. Conventional RL relies on trial-and-error interactions with the environment and reward feedback, which often leads to exploring unsafe operating regions and executing unsafe actions, especially when deployed in real-world power systems. To address these challenges, safe RL has been proposed to optimize operational objectives while ensuring safety constraints are met, keeping actions and states within safe regions throughout both training and deployment. Rather than relying solely on manually designed penalty terms for unsafe actions, as is common in conventional RL, safe RL methods reviewed here primarily leverage advanced and proactive mechanisms. These include techniques such as Lagrangian relaxation, safety layers, and theoretical guarantees like Lyapunov functions to rigorously enforce safety boundaries. This paper provides a comprehensive review of safe RL methods and their applications across various power system operations and control domains, including security control, real-time operation, operational planning, and emerging areas. It summarizes existing safe RL techniques, evaluates their performance, analyzes suitable deployment scenarios, and examines algorithm benchmarks and application environments. The paper also highlights real-world implementation cases and identifies critical challenges such as scalability in large-scale systems and robustness under uncertainty, providing potential solutions and outlining future directions to advance the reliable integration and deployment of safe RL in modern power systems.

References in corpus (22)

• A Brief Survey of Deep Reinforcement Learning
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• pandapower - an Open Source Python Tool for Convenient Modeling, Analysis and Optimization of Electric Power Systems
• PyPSA: Python for Power System Analysis
• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• Safe Reinforcement Learning Using Robust MPC
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• Strong SOCP Relaxations for the Optimal Power Flow Problem
• A Survey of Algorithms for Distributed Charging Control of Electric Vehicles in Smart Grid
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
• Exploring the Capabilities and Limitations of Large Language Models in the Electric Energy Sector
• PowerModelsDistribution.jl: An Open-Source Framework for Exploring Distribution Power Flow Formulations
• A Review of Safe Reinforcement Learning Methods for Modern Power Systems
• Multi-agent Deep Reinforcement Learning for Distributed Load Restoration
• A Tutorial on Meta-Reinforcement Learning
• Networked Multiagent Safe Reinforcement Learning for Low-carbon Demand Management in Distribution Network
• Constrained Update Projection Approach to Safe Policy Optimization
• A Safe Deep Reinforcement Learning Approach for Energy Efficient Federated Learning in Wireless Communication Networks
• Bridging Transient and Steady-State Performance in Voltage Control: A Reinforcement Learning Approach with Safe Gradient Flow
• SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents
• PowerSimulationsDynamics.jl -- An Open Source Modeling Package for Modern Power Systems with Inverter-Based Resources

Cited by in corpus (1)

• A Review of Safe Reinforcement Learning Methods for Modern Power Systems