Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization

Abstract

While reinforcement learning (RL) is gaining popularity in energy systems control, its real-world applications are limited due to the fact that the actions from learned policies may not satisfy functional requirements or be feasible for the underlying physical system. In this work, we propose PROjected Feasibility (PROF), a method to enforce convex operational constraints within neural policies. Specifically, we incorporate a differentiable projection layer within a neural network-based policy to enforce that all learned actions are feasible. We then update the policy end-to-end by propagating gradients through this differentiable projection layer, making the policy cognizant of the operational constraints. We demonstrate our method on two applications: energy-efficient building operation and inverter control. In the building operation setting, we show that PROF maintains thermal comfort requirements while improving energy efficiency by 4% over state-of-the-art methods. In the inverter control setting, PROF perfectly satisfies voltage constraints on the IEEE 37-bus feeder system, as it learns to curtail as little renewable energy as possible within its safety set.

arXiv commentAccepted at Twelfth ACM International Conference on Future Energy Systems (ACM e-Energy)

References in corpus (11)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Robust Adversarial Reinforcement Learning
• A Geometric Approach to Aggregate Flexibility Modeling of Thermostatically Controlled Loads
• Off-policy reinforcement learning for $H_{\infty }$ control design
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Deep Equilibrium Models
• Differentiable Convex Optimization Layers
• Neural Lyapunov Control
• High-Fidelity Machine Learning Approximations of Large-Scale Optimal Power Flow
• Verifiably Safe Off-Model Reinforcement Learning
• COHORT: Coordination of Heterogeneous Thermostatically Controlled Loads for Demand Flexibility

Cited by in corpus (4)

• Learning to Solve Optimization Problems with Hard Linear Constraints
• Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints
• CDCGen: Cross-Domain Conditional Generation via Normalizing Flows and Adversarial Training
• Leveraging Analytic Gradients in Provably Safe Reinforcement Learning