Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge

Abstract

Multiple automakers have in development or in production automated driving systems (ADS) that offer freeway-pilot functions. This type of ADS is typically limited to restricted-access freeways only, that is, the transition from manual to automated modes takes place only after the ramp merging process is completed manually. One major challenge to extend the automation to ramp merging is that the automated vehicle needs to incorporate and optimize long-term objectives (e.g. successful and smooth merge) when near-term actions must be safely executed. Moreover, the merging process involves interactions with other vehicles whose behaviors are sometimes hard to predict but may influence the merging vehicle optimal actions. To tackle such a complicated control problem, we propose to apply Deep Reinforcement Learning (DRL) techniques for finding an optimal driving policy by maximizing the long-term reward in an interactive environment. Specifically, we apply a Long Short-Term Memory (LSTM) architecture to model the interactive environment, from which an internal state containing historical driving information is conveyed to a Deep Q-Network (DQN). The DQN is used to approximate the Q-function, which takes the internal state as input and generates Q-values as output for action selection. With this DRL architecture, the historical impact of interactive environment on the long-term reward can be captured and taken into account for deciding the optimal control policy. The proposed architecture has the potential to be extended and applied to other autonomous driving scenarios such as driving through a complex intersection or changing lanes under varying traffic flow conditions.

IEEE International Conference on Intelligent Transportation Systems, Yokohama, Japan, 2017

References in corpus (2)

• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• End-to-End Deep Reinforcement Learning for Lane Keeping Assist

Cited by in corpus (20)

• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Behavioral decision-making for urban autonomous driving in the presence of pedestrians using Deep Recurrent Q-Network
• A Lane Merge Coordination Model for a V2X Scenario
• Autonomous Ramp Merge Maneuver Based on Reinforcement Learning with Continuous Action Space
• Driving Decision and Control for Autonomous Lane Change based on Deep Reinforcement Learning
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Fixed-Dimensional and Permutation Invariant State Representation of Autonomous Driving
• Quadratic Q-network for Learning Continuous Control for Autonomous Vehicles
• Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarios
• Deep Reinforcement Learning in Lane Merge Coordination for Connected Vehicles
• Dampen the Stop-and-Go Traffic with Connected and Automated Vehicles -- A Deep Reinforcement Learning Approach
• Network traffic instability in a two-ring system with automated driving and cooperative merging
• Optimal merging from an on-ramp into a high-speed lane dedicated to connected autonomous vehicles
• The Role of Machine Learning for Trajectory Prediction in Cooperative Driving
• Decision-Making under On-Ramp merge Scenarios by Distributional Soft Actor-Critic Algorithm
• An Agent-based Modelling Framework for Driving Policy Learning in Connected and Autonomous Vehicles
• Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving
• Towards Learning Generalizable Driving Policies from Restricted Latent Representations