GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning

Abstract

The heavy traffic congestion problem has always been a concern for modern cities. To alleviate traffic congestion, researchers use reinforcement learning (RL) to develop better traffic signal control (TSC) algorithms in recent years. However, most RL models are trained and tested in the same traffic flow environment, which results in a serious overfitting problem. Since the traffic flow environment in the real world keeps varying, these models can hardly be applied due to the lack of generalization ability. Besides, the limited number of accessible traffic flow data brings extra difficulty in testing the generalization ability of the models. In this paper, we design a novel traffic flow generator based on Wasserstein generative adversarial network to generate sufficient diverse and quality traffic flows and use them to build proper training and testing environments. Then we propose a meta-RL TSC framework GeneraLight to improve the generalization ability of TSC models. GeneraLight boosts the generalization performance by combining the idea of flow clustering and model-agnostic meta-learning. We conduct extensive experiments on multiple real-world datasets to show the superior performance of GeneraLight on generalizing to different traffic flows.

arXiv commentProceedings of the 29th ACM International on Conference on Information and Knowledge Management (CIKM). ACM, 2020

References in corpus (20)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• WaveNet: A Generative Model for Raw Audio
• A Simple Neural Attentive Meta-Learner
• Wasserstein GAN
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• CoLight: Learning Network-level Cooperation for Traffic Signal Control
• Robust Adversarial Reinforcement Learning
• Learning to reinforcement learn
• CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario
• A Survey on Traffic Signal Control Methods
• Meta-Reinforcement Learning of Structured Exploration Strategies
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Evolved Policy Gradients
• Hierarchically Structured Meta-learning
• Diversity is All You Need: Learning Skills without a Reward Function
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Unsupervised Meta-Learning for Reinforcement Learning
• Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning
• Toward Multimodal Model-Agnostic Meta-Learning

Cited by in corpus (2)

• LibSignal: An Open Library for Traffic Signal Control
• ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control