PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control

Abstract

This paper develops an efficient multi-agent deep reinforcement learning algorithm for cooperative controls in powergrids. Specifically, we consider the decentralized inverter-based secondary voltage control problem in distributed generators (DGs), which is first formulated as a cooperative multi-agent reinforcement learning (MARL) problem. We then propose a novel on-policy MARL algorithm, PowerNet, in which each agent (DG) learns a control policy based on (sub-)global reward but local states from its neighboring agents. Motivated by the fact that a local control from one agent has limited impact on agents distant from it, we exploit a novel spatial discount factor to reduce the effect from remote agents, to expedite the training process and improve scalability. Furthermore, a differentiable, learning-based communication protocol is employed to foster the collaborations among neighboring agents. In addition, to mitigate the effects of system uncertainty and random noise introduced during on-policy learning, we utilize an action smoothing factor to stabilize the policy execution. To facilitate training and evaluation, we develop PGSim, an efficient, high-fidelity powergrid simulation platform. Experimental results in two microgrid setups show that the developed PowerNet outperforms a conventional model-based control, as well as several state-of-the-art MARL algorithms. The decentralized learning scheme and high sample efficiency also make it viable to large-scale power grids.

arXiv comment11 pages

References in corpus (10)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Multi-agent Reinforcement Learning for Networked System Control

Cited by in corpus (1)

• Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic