Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning

Abstract

Finding the optimal signal timing strategy is a difficult task for the problem of large-scale traffic signal control (TSC). Multi-Agent Reinforcement Learning (MARL) is a promising method to solve this problem. However, there is still room for improvement in extending to large-scale problems and modeling the behaviors of other agents for each individual agent. In this paper, a new MARL, called Cooperative double Q-learning (Co-DQL), is proposed, which has several prominent features. It uses a highly scalable independent double Q-learning method based on double estimators and the UCB policy, which can eliminate the over-estimation problem existing in traditional independent Q-learning while ensuring exploration. It uses mean field approximation to model the interaction among agents, thereby making agents learn a better cooperative strategy. In order to improve the stability and robustness of the learning process, we introduce a new reward allocation mechanism and a local state sharing method. In addition, we analyze the convergence properties of the proposed algorithm. Co-DQL is applied on TSC and tested on a multi-traffic signal simulator. According to the results obtained on several traffic scenarios, Co- DQL outperforms several state-of-the-art decentralized MARL algorithms. It can effectively shorten the average waiting time of the vehicles in the whole road system.

14 pages, 11 figures

References in corpus (1)

• FeUdal Networks for Hierarchical Reinforcement Learning

Cited by in corpus (11)

• IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control
• Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control
• Towards Multi-agent Reinforcement Learning based Traffic Signal Control through Spatio-temporal Hypergraphs
• ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control
• Multi-Agent Trust Region Policy Optimization
• Multi-Agent Deep Reinforcement Learning for Request Dispatching in Distributed-Controller Software-Defined Networking
• On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)
• Traffic Co-Simulation Framework Empowered by Infrastructure Camera Sensing and Reinforcement Learning
• Optimizing Large-Scale Fleet Management on a Road Network using Multi-Agent Deep Reinforcement Learning with Graph Neural Network
• Multi-intersection Traffic Optimisation: A Benchmark Dataset and a Strong Baseline
• Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules