IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control

Abstract

Scaling adaptive traffic-signal control involves dealing with combinatorial state and action spaces. Multi-agent reinforcement learning attempts to address this challenge by distributing control to specialized agents. However, specialization hinders generalization and transferability, and the computational graphs underlying neural-networks architectures -- dominating in the multi-agent setting -- do not offer the flexibility to handle an arbitrary number of entities which changes both between road networks, and over time as vehicles traverse the network. We introduce Inductive Graph Reinforcement Learning (IG-RL) based on graph-convolutional networks which adapts to the structure of any road network, to learn detailed representations of traffic-controllers and their surroundings. Our decentralized approach enables learning of a transferable-adaptive-traffic-signal-control policy. After being trained on an arbitrary set of road networks, our model can generalize to new road networks, traffic distributions, and traffic regimes, with no additional training and a constant number of parameters, enabling greater scalability compared to prior methods. Furthermore, our approach can exploit the granularity of available data by capturing the (dynamic) demand at both the lane and the vehicle levels. The proposed method is tested on both road networks and traffic settings never experienced during training. We compare IG-RL to multi-agent reinforcement learning and domain-specific baselines. In both synthetic road networks and in a larger experiment involving the control of the 3,971 traffic signals of Manhattan, we show that different instantiations of IG-RL outperform baselines.

11 pages, 10 figures, 1 table. IEEE Transactions on Intelligent Transportation Systems (2021)

References in corpus (7)

• Semi-Supervised Classification with Graph Convolutional Networks
• Solving Rubik's Cube with a Robot Hand
• CoLight: Learning Network-level Cooperation for Traffic Signal Control
• Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning
• Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control

Cited by in corpus (2)

• LibSignal: An Open Library for Traffic Signal Control
• Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control