Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control

Abstract

This study compares Deep Reinforcement Learning (DRL) and Model Predictive Control (MPC) for Adaptive Cruise Control (ACC) design in car-following scenarios. A first-order system is used as the Control-Oriented Model (COM) to approximate the acceleration command dynamics of a vehicle. Based on the equations of the control system and the multi-objective cost function, we train a DRL policy using Deep Deterministic Policy Gradient (DDPG) and solve the MPC problem via Interior-Point Optimization (IPO). Simulation results for the episode costs show that, when there are no modeling errors and the testing inputs are within the training data range, the DRL solution is equivalent to MPC with a sufficiently long prediction horizon. Particularly, the DRL episode cost is only 5.8% higher than the benchmark solution provided by optimizing the entire episode via IPO. The DRL control performance degrades when the testing inputs are outside the training data range, indicating inadequate generalization. When there are modeling errors due to control delays, disturbances, and/or testing with a High-Fidelity Model (HFM) of the vehicle, the DRL-trained policy performs better with large modeling errors while having similar performance as MPC when the modeling errors are small.

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Trust Region Policy Optimization
• Deep Reinforcement Learning that Matters
• Data-driven Economic NMPC using Reinforcement Learning
• Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
• Learning values across many orders of magnitude
• Adaptive Tube-based Nonlinear MPC for Ecological Autonomous Cruise Control of Plug-in Hybrid Electric Vehicles
• How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learning
• Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles
• WiseMove: A Framework for Safe Deep Reinforcement Learning for Autonomous Driving

Cited by in corpus (14)

• Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Driving
• Hybrid Car-Following Strategy based on Deep Deterministic Policy Gradient and Cooperative Adaptive Cruise Control
• Autonomous Platoon Control with Integrated Deep Reinforcement Learning and Dynamic Programming
• Relaxed Actor-Critic with Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems
• Formulation and validation of a car-following model based on deep reinforcement learning
• Data Driven Control with Learned Dynamics: Model-Based versus Model-Free Approach
• A Deep Reinforcement Learning Based Motion Cueing Algorithm for Vehicle Driving Simulation
• AVDDPG: Federated reinforcement learning applied to autonomous platoon control
• Anti-Jerk On-Ramp Merging Using Deep Reinforcement Learning
• Neural Network Based Model Predictive Control for an Autonomous Vehicle
• Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships
• Adaptive Estimation-Based Safety-Critical Cruise Control of Vehicular Platoons
• Tactical Decision Making for Autonomous Trucks by Deep Reinforcement Learning with Total Cost of Operation Based Reward
• Learning-based synthesis of robust linear time-invariant controllers