A Survey of Deep Learning Techniques for Autonomous Driving

Abstract

The last decade witnessed increasingly rapid progress in self-driving vehicle technology, mainly backed up by advances in the area of deep learning and artificial intelligence. The objective of this paper is to survey the current state-of-the-art on deep learning technologies used in autonomous driving. We start by presenting AI-based self-driving architectures, convolutional and recurrent neural networks, as well as the deep reinforcement learning paradigm. These methodologies form a base for the surveyed driving scene perception, path planning, behavior arbitration and motion control algorithms. We investigate both the modular perception-planning-action pipeline, where each module is built using deep learning methods, as well as End2End systems, which directly map sensory information to steering commands. Additionally, we tackle current challenges encountered in designing AI architectures for autonomous driving, such as their safety, training data sources and computational hardware. The comparison presented in this survey helps to gain insight into the strengths and limitations of deep learning and AI approaches for autonomous driving and assist with design choices

28 pages, 7 figures. arXiv admin note: text overlap with arXiv:1709.02435, arXiv:1610.01256 by other authors

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SSD: Single Shot MultiBox Detector
• Continuous control with deep reinforcement learning
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• YOLOv3: An Incremental Improvement
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• End to End Learning for Self-Driving Cars
• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Deep Reinforcement Learning framework for Autonomous Driving
• Domain Adaptation for Statistical Classifiers
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• Continuous Deep Q-Learning with Model-based Acceleration
• Learning Dexterous In-Hand Manipulation
• Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
• VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry
• Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
• An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software
• End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies
• Learning to Localize Using a LiDAR Intensity Map
• Learning to See the Wood for the Trees: Deep Laser Localization in Urban and Natural Environments on a CPU
• End-to-End Tracking and Semantic Segmentation Using Recurrent Neural Networks
• Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization
• Path Tracking of Highly Dynamic Autonomous Vehicle Trajectories via Iterative Learning Control
• Aggressive Deep Driving: Model Predictive Control with a CNN Cost Model
• Debugging Machine Learning Tasks
• Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Search
• Feature Analysis and Selection for Training an End-to-End Autonomous Vehicle Controller Using the Deep Learning Approach

Cited by in corpus (129)

• Machine learning and deep learning
• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Partially Observable Markov Decision Processes in Robotics: A Survey
• Theory of overparametrization in quantum neural networks
• Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey
• Neural Network Potentials for Chemistry: Concepts, Applications and Prospects
• Optical coherent dot-product chip for sophisticated deep learning regression
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• A Tutorial on 5G Positioning
• A survey of image labelling for computer vision applications
• Learning Collision-Free Space Detection from Stereo Images: Homography Matrix Brings Better Data Augmentation
• Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Towards Training Reproducible Deep Learning Models
• Fully Autonomous Programming with Large Language Models
• Explainable Artificial Intelligence (XAI): An Engineering Perspective
• NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks
• Machine learning in spectral domain
• Automated machine learning for secure key rate in discrete-modulated continuous-variable quantum key distribution
• Energy-based error bound of physics-informed neural network solutions in elasticity
• New Trends in Quantum Machine Learning
• DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• High-fidelity Prediction of Megapixel Longitudinal Phase-space Images of Electron Beams using Encoder-Decoder Neural Networks
• Organization of machine learning based product development as per ISO 26262 and ISO/PAS 21448
• Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing
• Deep Reinforcement Learning for Human-Like Driving Policies in Collision Avoidance Tasks of Self-Driving Cars
• Lightweight Residual Densely Connected Convolutional Neural Network
• HOPE: A Reinforcement Learning-based Hybrid Policy Path Planner for Diverse Parking Scenarios
• Quantum Transfer Learning with Adversarial Robustness for Classification of High-Resolution Image Datasets
• Cloud2Edge Elastic AI Framework for Prototyping and Deployment of AI Inference Engines in Autonomous Vehicles
• Robust Reinforcement Learning-based Autonomous Driving Agent for Simulation and Real World
• Spectral pruning of fully connected layers: ranking the nodes based on the eigenvalues
• A Survey on Autonomous Vehicle Control in the Era of Mixed-Autonomy: From Physics-Based to AI-Guided Driving Policy Learning
• Graph-Segmenter: Graph Transformer with Boundary-aware Attention for Semantic Segmentation
• Single Shot MC Dropout Approximation
• Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysis
• A Framework for Understanding AI-Induced Field Change: How AI Technologies are Legitimized and Institutionalized
• Reachability Analysis and Safety Verification of Neural Feedback Systems via Hybrid Zonotopes
• Centimeter-level Positioning by Instantaneous Lidar-aided GNSS Ambiguity Resolution
• On the training of sparse and dense deep neural networks: less parameters, same performance
• Two Souls in an Adversarial Image: Towards Universal Adversarial Example Detection using Multi-view Inconsistency
• Deep learning for nano-photonic materials -- The solution to everything!?
• Knowledge-based Entity Prediction for Improved Machine Perception in Autonomous Systems
• Neural Insights for Digital Marketing Content Design
• Vehicle Suspension Recommendation System: Multi-Fidelity Neural Network-based Mechanism Design Optimization
• LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
• CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection
• Evaluating Object (mis)Detection from a Safety and Reliability Perspective: Discussion and Measures
• LExCI: A Framework for Reinforcement Learning with Embedded Systems
• OctoPath: An OcTree Based Self-Supervised Learning Approach to Local Trajectory Planning for Mobile Robots
• RoHNAS: A Neural Architecture Search Framework with Conjoint Optimization for Adversarial Robustness and Hardware Efficiency of Convolutional and Capsule Networks
• CyberCortex.AI: An AI-based Operating System for Autonomous Robotics and Complex Automation
• On the Road with 16 Neurons: Mental Imagery with Bio-inspired Deep Neural Networks
• Recurrent Spectral Network (RSN): shaping the basin of attraction of a discrete map to reach automated classification
• Visual Analytics of Neuron Vulnerability to Adversarial Attacks on Convolutional Neural Networks
• Deep Neural Networks for Computational Optical Form Measurements
• Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis
• Towards Dependable Autonomous Systems Based on Bayesian Deep Learning Components
• On the Connection between Game-Theoretic Feature Attributions and Counterfactual Explanations
• Study on the Helpfulness of Explainable Artificial Intelligence
• Effective Shortcut Technique for GAN
• Efficient Perception, Planning, and Control Algorithm for Vision-Based Automated Vehicles
• Aleatoric uncertainty for Errors-in-Variables models in deep regression
• NBA: defensive distillation for backdoor removal via neural behavior alignment
• Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
• Cycle Consistency-based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems
• Safe Autonomous Racing via Approximate Reachability on Ego-vision
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• AI based Algorithms of Path Planning, Navigation and Control for Mobile Ground Robots and UAVs
• Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI
• How Could Generative AI Support Compliance with the EU AI Act? A Review for Safe Automated Driving Perception
• Towards Robust Neural Networks via Close-loop Control
• Preserve, Promote, or Attack? GNN Explanation via Topology Perturbation
• Action-Based Representation Learning for Autonomous Driving
• Learning Autonomous Race Driving with Action Mapping Reinforcement Learning
• Improving Robustness of Deep Neural Networks for Aerial Navigation by Incorporating Input Uncertainty
• Gradient Flow Based Phase-Field Modeling Using Separable Neural Networks
• PhysORD: A Neuro-Symbolic Approach for Physics-infused Motion Prediction in Off-road Driving
• JaxSGMC: Modular stochastic gradient MCMC in JAX
• A Comparison for Patch-level Classification of Deep Learning Methods on Transparent Environmental Microorganism Images: from Convolutional Neural Networks to Visual Transformers
• Exploiting Inter-Sample Information for Long-tailed Out-of-Distribution Detection
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• 4C: A Computation, Communication, and Control Co-Design Framework for CAVs
• BCNet: A Deep Convolutional Neural Network for Breast Cancer Grading
• Active Digital Twins via Active Inference
• Analysis of memory consumption by neural networks based on hyperparameters
• Enhancing Autonomous Vehicle Navigation with a Clothoid-Based Lateral Controller
• Corner case data description and detection
• Inverse RL Scene Dynamics Learning for Nonlinear Predictive Control in Autonomous Vehicles
• Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data
• A Picture is Worth a Collaboration: Accumulating Design Knowledge for Computer-Vision-based Hybrid Intelligence Systems
• SIGVerse: A cloud-based VR platform for research on social and embodied human-robot interaction
• DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction
• CostNet: An End-to-End Framework for Goal-Directed Reinforcement Learning
• Prediction-Based Reachability for Collision Avoidance in Autonomous Driving
• Physics-Inspired Interpretability Of Machine Learning Models
• Mic-hackathon 2024: Hackathon on Machine Learning for Electron and Scanning Probe Microscopy
• A Statistician Teaches Deep Learning
• Textual Backdoor Attacks Can Be More Harmful via Two Simple Tricks
• A Novel Data Encryption Method Inspired by Adversarial Attacks
• Towards the Characterization of Representations Learned via Capsule-based Network Architectures
• Structure Information is the Key: Self-Attention RoI Feature Extractor in 3D Object Detection
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer
• Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles
• Cosmic velocity, density and halo mass function: Insights from deep learning
• Yaw-Guided Imitation Learning for Autonomous Driving in Urban Environments
• Learning by Cheating : An End-to-End Zero Shot Framework for Autonomous Drone Navigation
• Training Efficiency and Robustness in Deep Learning
• Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects
• Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting
• Reachability Is NP-Complete Even for the Simplest Neural Networks
• Robust Unsupervised Multi-Object Tracking in Noisy Environments
• VeniBot: Towards Autonomous Venipuncture with Semi-supervised Vein Segmentation from Ultrasound Images
• DeepWORD: A GCN-based Approach for Owner-Member Relationship Detection in Autonomous Driving
• Coverage-Guided Road Selection and Prioritization for Efficient Testing in Autonomous Driving Systems
• Dependable Neural Networks for Safety Critical Tasks
• Confidential Machine Learning on Untrusted Platforms: A Survey
• End to End Binarized Neural Networks for Text Classification