Unified Automatic Control of Vehicular Systems with Reinforcement Learning

Abstract

Emerging vehicular systems with increasing proportions of automated components present opportunities for optimal control to mitigate congestion and increase efficiency. There has been a recent interest in applying deep reinforcement learning (DRL) to these nonlinear dynamical systems for the automatic design of effective control strategies. Despite conceptual advantages of DRL being model-free, studies typically nonetheless rely on training setups that are painstakingly specialized to specific vehicular systems. This is a key challenge to efficient analysis of diverse vehicular and mobility systems. To this end, this article contributes a streamlined methodology for vehicular microsimulation and discovers high performance control strategies with minimal manual design. A variable-agent, multi-task approach is presented for optimization of vehicular Partially Observed Markov Decision Processes. The methodology is experimentally validated on mixed autonomy traffic systems, where fractions of vehicles are automated; empirical improvement, typically 15-60% over a human driving baseline, is observed in all configurations of six diverse open or closed traffic systems. The study reveals numerous emergent behaviors resembling wave mitigation, traffic signaling, and ramp metering. Finally, the emergent behaviors are analyzed to produce interpretable control strategies, which are validated against the learned control strategies.

arXiv comment16 pages, 14 figures, IEEE Transactions on Automation Science and Engineering (T-ASE), 2022

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Congested Traffic States in Empirical Observations and Microscopic Simulations
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments
• Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks
• CoLight: Learning Network-level Cooperation for Traffic Signal Control
• RLlib: Abstractions for Distributed Reinforcement Learning
• Flow: A Modular Learning Framework for Mixed Autonomy Traffic
• Reinforcement Learning for Mixed Autonomy Intersections
• Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles
• Optimizing Mixed Autonomy Traffic Flow With Decentralized Autonomous Vehicles and Multi-Agent RL

Cited by in corpus (3)

• Learning to Control and Coordinate Mixed Traffic Through Robot Vehicles at Complex and Unsignalized Intersections
• Vehicle Sequencing at Signal-Free Intersections: Analytical Performance Guarantees Based on PDMP Formulation
• Cooperative Advisory Residual Policies for Congestion Mitigation