A Survey of Deep RL and IL for Autonomous Driving Policy Learning

Abstract

Autonomous driving (AD) agents generate driving policies based on online perception results, which are obtained at multiple levels of abstraction, e.g., behavior planning, motion planning and control. Driving policies are crucial to the realization of safe, efficient and harmonious driving behaviors, where AD agents still face substantial challenges in complex scenarios. Due to their successful application in fields such as robotics and video games, the use of deep reinforcement learning (DRL) and deep imitation learning (DIL) techniques to derive AD policies have witnessed vast research efforts in recent years. This paper is a comprehensive survey of this body of work, which is conducted at three levels: First, a taxonomy of the literature studies is constructed from the system perspective, among which five modes of integration of DRL/DIL models into an AD architecture are identified. Second, the formulations of DRL/DIL models for conducting specified AD tasks are comprehensively reviewed, where various designs on the model state and action spaces and the reinforcement learning rewards are covered. Finally, an in-depth review is conducted on how the critical issues of AD applications regarding driving safety, interaction with other traffic participants and uncertainty of the environment are addressed by the DRL/DIL models. To the best of our knowledge, this is the first survey to focus on AD policy learning using DRL/DIL, which is addressed simultaneously from the system, task-driven and problem-driven perspectives. We share and discuss findings, which may lead to the investigation of various topics in the future.

References in corpus (52)

• Continuous control with deep reinforcement learning
• A Brief Survey of Deep Reinforcement Learning
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Prioritized Experience Replay
• Soft Actor-Critic Algorithms and Applications
• A Survey of Deep Learning Techniques for Autonomous Driving
• Dueling Network Architectures for Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Deep Bayesian Active Learning with Image Data
• Deep Reinforcement Learning: An Overview
• An Algorithmic Perspective on Imitation Learning
• Continuous Deep Q-Learning with Model-based Acceleration
• A Survey of End-to-End Driving: Architectures and Training Methods
• CARLA: An Open Urban Driving Simulator
• Maximum Entropy Deep Inverse Reinforcement Learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Reinforced Continual Learning
• Driving Policy Transfer via Modularity and Abstraction
• Distributional Reinforcement Learning with Quantile Regression
• Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge
• Learning human behaviors from motion capture by adversarial imitation
• End-to-End Deep Reinforcement Learning for Lane Keeping Assist
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• Controlling an Autonomous Vehicle with Deep Reinforcement Learning
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Learning to Drive using Inverse Reinforcement Learning and Deep Q-Networks
• End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies
• Towards a Functional System Architecture for Automated Vehicles
• Combining Deep Reinforcement Learning and Safety Based Control for Autonomous Driving
• Robust Imitation of Diverse Behaviors
• Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States
• Dynamic Input for Deep Reinforcement Learning in Autonomous Driving
• Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
• Deep Q-Network Based Decision Making for Autonomous Driving
• Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning
• Reinforcement Learning with Probabilistic Guarantees for Autonomous Driving
• Safe end-to-end imitation learning for model predictive control
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Lane-Merging Using Policy-based Reinforcement Learning and Post-Optimization
• Experience-Based Heuristic Search: Robust Motion Planning with Deep Q-Learning
• Interaction-aware Decision Making with Adaptive Strategies under Merging Scenarios
• Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective

Cited by in corpus (9)

• HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving
• Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigation
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• Arena-Bench: A Benchmarking Suite for Obstacle Avoidance Approaches in Highly Dynamic Environments
• Cooperative Behavior Planning for Automated Driving using Graph Neural Networks
• Modeling Interactions of Autonomous Vehicles and Pedestrians with Deep Multi-Agent Reinforcement Learning for Collision Avoidance
• Automatic Intersection Management in Mixed Traffic Using Reinforcement Learning and Graph Neural Networks
• Integration of Reinforcement Learning Based Behavior Planning With Sampling Based Motion Planning for Automated Driving
• An Enhanced Graph Representation for Machine Learning Based Automatic Intersection Management