Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges

Abstract

With large-scale integration of renewable generation and distributed energy resources, modern power systems are confronted with new operational challenges, such as growing complexity, increasing uncertainty, and aggravating volatility. Meanwhile, more and more data are becoming available owing to the widespread deployment of smart meters, smart sensors, and upgraded communication networks. As a result, data-driven control techniques, especially reinforcement learning (RL), have attracted surging attention in recent years. This paper provides a comprehensive review of various RL techniques and how they can be applied to decision-making and control in power systems. In particular, we select three key applications, i.e., frequency regulation, voltage control, and energy management, as examples to illustrate RL-based models and solutions. We then present the critical issues in the application of RL, i.e., safety, robustness, scalability, and data. Several potential future directions are discussed as well.

References in corpus (16)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Robust Adversarial Reinforcement Learning
• Optimization for deep learning: theory and algorithms
• Robust Deep Reinforcement Learning with Adversarial Attacks
• Online Scheduling of a Residential Microgrid via Monte-Carlo Tree Search and a Learned Model
• Gym-ANM: Reinforcement Learning Environments for Active Network Management Tasks in Electricity Distribution Systems
• Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism
• Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation
• pymgrid: An Open-Source Python Microgrid Simulator for Applied Artificial Intelligence Research
• Active Reinforcement Learning: Observing Rewards at a Cost
• Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach
• PowerGym: A Reinforcement Learning Environment for Volt-Var Control in Power Distribution Systems
• Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training
• Scalable Voltage Control using Structure-Driven Hierarchical Deep Reinforcement Learning
• Bi-level Off-policy Reinforcement Learning for Volt/VAR Control Involving Continuous and Discrete Devices
• Stability Constrained Reinforcement Learning for Real-Time Voltage Control

Cited by in corpus (15)

• Fusion of Model-free Reinforcement Learning with Microgrid Control: Review and Vision
• A Review of Safe Reinforcement Learning Methods for Modern Power Systems
• Resilient Control of Networked Microgrids using Vertical Federated Reinforcement Learning: Designs and Real-Time Test-Bed Validations
• Reinforcement Learning for Distributed Transient Frequency Control with Stability and Safety Guarantees
• A Survey of Reinforcement Learning for Optimization in Automation
• Bridging Transient and Steady-State Performance in Voltage Control: A Reinforcement Learning Approach with Safe Gradient Flow
• An Artificial Neural Network based approach for Harmonic Component Prediction in a Distribution Line
• Data-Driven Linear Koopman Embedding for Networked Systems: Model-Predictive Grid Control
• Contingency-constrained economic dispatch with safe reinforcement learning
• Distributionally Robust Safety Filter for Learning-Based Control in Active Distribution Systems
• Temporal Prototype-Aware Learning for Active Voltage Control on Power Distribution Networks
• Optimization-Based Exploration of the Feasible Power Flow Space for Rapid Data Collection
• CommonPower: A Framework for Safe Data-Driven Smart Grid Control
• Constrained Policy Optimization for Stochastic Optimal Control under Nonstationary Uncertainties
• On Linear Quadratic Potential Games