Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies

Abstract

Since DARPA Grand Challenges (rural) in 2004/05 and Urban Challenges in 2007, autonomous driving has been the most active field of AI applications. Almost at the same time, deep learning has made breakthrough by several pioneers, three of them (also called fathers of deep learning), Hinton, Bengio and LeCun, won ACM Turin Award in 2019. This is a survey of autonomous driving technologies with deep learning methods. We investigate the major fields of self-driving systems, such as perception, mapping and localization, prediction, planning and control, simulation, V2X and safety etc. Due to the limited space, we focus the analysis on several key areas, i.e. 2D and 3D object detection in perception, depth estimation from cameras, multiple sensor fusion on the data, feature and task level respectively, behavior modelling and prediction of vehicle driving and pedestrian trajectories.

References in corpus (63)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• YOLOv4: Optimal Speed and Accuracy of Object Detection
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• End to End Learning for Self-Driving Cars
• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
• A Survey of Autonomous Driving: Common Practices and Emerging Technologies
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• A Survey of Deep Learning-based Object Detection
• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• FCOS: Fully Convolutional One-Stage Object Detection
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Deep Learning in Video Multi-Object Tracking: A Survey
• Object Detection in 20 Years: A Survey
• Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods
• Deep Continuous Fusion for Multi-Sensor 3D Object Detection
• Deep Learning: A Critical Appraisal
• Xception: Deep Learning with Depthwise Separable Convolutions
• SfM-Net: Learning of Structure and Motion from Video
• Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving
• Confidence Propagation through CNNs for Guided Sparse Depth Regression
• A Survey on Neural Architecture Search
• Meta-Learning: A Survey
• Learning a Driving Simulator
• End-to-end Learning of Multi-sensor 3D Tracking by Detection
• Optimization for deep learning: theory and algorithms
• IPOD: Intensive Point-based Object Detector for Point Cloud
• Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving
• StarNet: Targeted Computation for Object Detection in Point Clouds
• Selective Kernel Networks
• Orthographic Feature Transform for Monocular 3D Object Detection
• Pyramid Stereo Matching Network
• Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Loss
• Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry
• Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States
• Region Proposal by Guided Anchoring
• STD: Sparse-to-Dense 3D Object Detector for Point Cloud
• Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network
• RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving
• UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning
• VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
• RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Driving
• Voxel-FPN: multi-scale voxel feature aggregation in 3D object detection from point clouds
• CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection
• StarNet: Pedestrian Trajectory Prediction using Deep Neural Network in Star Topology
• LEGO: Learning Edge with Geometry all at Once by Watching Videos
• Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
• Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving
• Segmentation is All You Need
• A Hierarchical Architecture for Sequential Decision-Making in Autonomous Driving using Deep Reinforcement Learning
• An Auto-tuning Framework for Autonomous Vehicles
• Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry
• Self-Supervised Learning of Depth and Ego-motion with Differentiable Bundle Adjustment
• Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning
• Multi-Sensor 3D Object Box Refinement for Autonomous Driving
• Class-specific Anchoring Proposal for 3D Object Recognition in LIDAR and RGB Images
• Unsupervised Video Depth Estimation Based on Ego-motion and Disparity Consensus
• Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation

Cited by in corpus (4)

• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys
• Deep Deterministic Uncertainty for Semantic Segmentation
• Reliving the Dataset: Combining the Visualization of Road Users' Interactions with Scenario Reconstruction in Virtual Reality
• Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning