A Survey of End-to-End Driving: Architectures and Training Methods

Abstract

Autonomous driving is of great interest to industry and academia alike. The use of machine learning approaches for autonomous driving has long been studied, but mostly in the context of perception. In this paper we take a deeper look on the so called end-to-end approaches for autonomous driving, where the entire driving pipeline is replaced with a single neural network. We review the learning methods, input and output modalities, network architectures and evaluation schemes in end-to-end driving literature. Interpretability and safety are discussed separately, as they remain challenging for this approach. Beyond providing a comprehensive overview of existing methods, we conclude the review with an architecture that combines the most promising elements of the end-to-end autonomous driving systems.

References in corpus (26)

• Conditional Generative Adversarial Nets
• Explaining and Harnessing Adversarial Examples
• How transferable are features in deep neural networks?
• Unsupervised Domain Adaptation by Backpropagation
• A Survey of Autonomous Driving: Common Practices and Emerging Technologies
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Dota 2 with Large Scale Deep Reinforcement Learning
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Multimodal End-to-End Autonomous Driving
• CARLA: An Open Urban Driving Simulator
• IntentNet: Learning to Predict Intention from Raw Sensor Data
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset
• Learning a Driving Simulator
• Causal Confusion in Imitation Learning
• Variational End-to-End Navigation and Localization
• DDD17: End-To-End DAVIS Driving Dataset
• Does computer vision matter for action?
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Efficient Black-box Assessment of Autonomous Vehicle Safety
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?
• A Commute in Data: The comma2k19 Dataset
• Learning from Maps: Visual Common Sense for Autonomous Driving
• Aggressive Deep Driving: Model Predictive Control with a CNN Cost Model
• Learning Accurate, Comfortable and Human-like Driving

Cited by in corpus (32)

• Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives
• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review
• Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving
• BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems
• End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent
• LiDAR-Based Place Recognition For Autonomous Driving: A Survey
• Control Strategies for Autonomous Vehicles
• Adversarial Driving: Attacking End-to-End Autonomous Driving
• Computing Systems for Autonomous Driving: State-of-the-Art and Challenges
• SAGE: A Split-Architecture Methodology for Efficient End-to-End Autonomous Vehicle Control
• Navigation-Oriented Scene Understanding for Robotic Autonomy: Learning to Segment Driveability in Egocentric Images
• FRENETIX: A High-Performance and Modular Motion Planning Framework for Autonomous Driving
• All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes
• Autonomous Driving Small-Scale Cars: A Survey of Recent Development
• CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving
• Navigation In Urban Environments Amongst Pedestrians Using Multi-Objective Deep Reinforcement Learning
• Improving the Generalization of End-to-End Driving through Procedural Generation
• Efficient Perception, Planning, and Control Algorithm for Vision-Based Automated Vehicles
• Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning
• GC-GAT: Multimodal Vehicular Trajectory Prediction using Graph Goal Conditioning and Cross-context Attention
• Safe Trajectory Planning Using Reinforcement Learning for Self Driving
• Pedestrian-Aware Motion Planning for Autonomous Driving in Complex Urban Scenarios
• Behavioral Research and Practical Models of Drivers' Attention
• A Mapping of Assurance Techniques for Learning Enabled Autonomous Systems to the Systems Engineering Lifecycle
• Combating the effects of speed and delays in end-to-end self-driving
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• What data do we need for training an AV motion planner?
• Yaw-Guided Imitation Learning for Autonomous Driving in Urban Environments
• DeepIPCv2: LiDAR-powered Robust Environmental Perception and Navigational Control for Autonomous Vehicle
• A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation