Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Traffic

Abstract

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL), a powerful data-driven control method, has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic network (MA2C) is developed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is proposed to incorporate fuel efficiency, driving comfort, and safety of autonomous driving. Comprehensive experimental results, conducted under three different traffic densities and various levels of human driver aggressiveness, show that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety and driver comfort.

arXiv commentThis paper was published on Autonomous Intelligent Systems (Volume 2, article number 5, 2022)

References in corpus (15)

• Semi-Supervised Classification with Graph Convolutional Networks
• Playing Atari with Deep Reinforcement Learning
• Congested Traffic States in Empirical Observations and Microscopic Simulations
• Trust Region Policy Optimization
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Connectivity statistics of store-and-forward inter-vehicle communication
• Multi-agent Reinforcement Learning for Networked System Control
• A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network
• Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic
• A Cooperative Control Framework for CAV Lane Change in a Mixed Traffic Environment
• Leveraging the Capabilities of Connected and Autonomous Vehicles and Multi-Agent Reinforcement Learning to Mitigate Highway Bottleneck Congestion
• Parameter Sharing Reinforcement Learning Architecture for Multi Agent Driving Behaviors
• Towards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning
• Efficient Motion Planning for Automated Lane Change based on Imitation Learning and Mixed-Integer Optimization

Cited by in corpus (6)

• Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors
• Stochastic Time-Optimal Trajectory Planning for Connected and Automated Vehicles in Mixed-Traffic Merging Scenarios
• IntNet: A Communication-Driven Multi-Agent Reinforcement Learning Framework for Cooperative Autonomous Driving
• Counterfactual experience augmented off-policy reinforcement learning