Gym-ANM: Reinforcement Learning Environments for Active Network Management Tasks in Electricity Distribution Systems

Abstract

Active network management (ANM) of electricity distribution networks include many complex stochastic sequential optimization problems. These problems need to be solved for integrating renewable energies and distributed storage into future electrical grids. In this work, we introduce Gym-ANM, a framework for designing reinforcement learning (RL) environments that model ANM tasks in electricity distribution networks. These environments provide new playgrounds for RL research in the management of electricity networks that do not require an extensive knowledge of the underlying dynamics of such systems. Along with this work, we are releasing an implementation of an introductory toy-environment, ANM6-Easy, designed to emphasize common challenges in ANM. We also show that state-of-the-art RL algorithms can already achieve good performance on ANM6-Easy when compared against a model predictive control (MPC) approach. Finally, we provide guidelines to create new Gym-ANM environments differing in terms of (a) the distribution network topology and parameters, (b) the observation space, (c) the modelling of the stochastic processes present in the system, and (d) a set of hyperparameters influencing the reward signal. Gym-ANM can be downloaded at https://github.com/robinhenry/gym-anm.

arXiv comment15 main pages, 17 pages of appendix, 10 figures, GitHub repository: https://github.com/robinhenry/gym-anm

References in corpus (7)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Deep Reinforcement Learning framework for Autonomous Driving
• On the Complexity of Best Arm Identification in Multi-Armed Bandit Models
• OpenAI Gym
• CVXPY: A Python-Embedded Modeling Language for Convex Optimization
• Challenges of Real-World Reinforcement Learning

Cited by in corpus (9)

• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• Learning to run a Power Network Challenge: a Retrospective Analysis
• TreeC: a method to generate interpretable energy management systems using a metaheuristic algorithm
• Optimizing Energy Management of Smart Grid using Reinforcement Learning aided by Surrogate models built using Physics-informed Neural Networks
• Gym-ANM: Open-source software to leverage reinforcement learning for power system management in research and education
• Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym
• CommonPower: A Framework for Safe Data-Driven Smart Grid Control
• Understanding the Safety Requirements for Learning-based Power Systems Operations
• Gym-TORAX: Open-source software for integrating reinforcement learning with plasma control simulators in tokamak research