Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

Abstract

Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this paper devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Spectral Networks and Locally Connected Networks on Graphs
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Deep Convolutional Networks on Graph-Structured Data
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Deep Continuous Fusion for Multi-Sensor 3D Object Detection
• Deep Parametric Continuous Convolutional Neural Networks
• PointSIFT: A SIFT-like Network Module for 3D Point Cloud Semantic Segmentation
• CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks
• Self-Supervised Model Adaptation for Multimodal Semantic Segmentation
• Confidence Propagation through CNNs for Guided Sparse Depth Regression
• Progressive LiDAR Adaptation for Road Detection
• End-to-end Learning of Multi-sensor 3D Tracking by Detection
• Track to Reconstruct and Reconstruct to Track
• IPOD: Intensive Point-based Object Detector for Point Cloud
• 3D Object Detection Using Scale Invariant and Feature Reweighting Networks
• Deep Unsupervised Learning of 3D Point Clouds via Graph Topology Inference and Filtering
• 3D-BEVIS: Bird's-Eye-View Instance Segmentation
• Learning Guided Convolutional Network for Depth Completion
• From Here to There: Video Inbetweening Using Direct 3D Convolutions
• Identifying Unknown Instances for Autonomous Driving
• Multi-view PointNet for 3D Scene Understanding
• LiStereo: Generate Dense Depth Maps from LIDAR and Stereo Imagery

Cited by in corpus (23)

• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys
• Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
• Milestones in Autonomous Driving and Intelligent Vehicles Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviors
• Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving
• End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent
• Benchmarking Robustness of AI-Enabled Multi-sensor Fusion Systems: Challenges and Opportunities
• LiDAR-guided Stereo Matching with a Spatial Consistency Constraint
• Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion
• Multi-Modal 3D Object Detection in Autonomous Driving: a Survey
• MultiTest: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systems
• Simultaneous Localization and Mapping Related Datasets: A Comprehensive Survey
• PointSmile: Point Self-supervised Learning via Curriculum Mutual Information
• VLCQ: Post-training quantization for deep neural networks using variable length coding
• Uncertainty-Aware Voxel based 3D Object Detection and Tracking with von-Mises Loss
• All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Towards Sensor Data Abstraction of Autonomous Vehicle Perception Systems
• An Expert Ensemble for Detecting Anomalous Scenes, Interactions, and Behaviors in Autonomous Driving
• PRISE: Privacy-pReserving Image Searchable Encryption Scheme for Intelligent Vehicle Systems
• SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images
• LiDAR Point Cloud Colourisation Using Multi-Camera Fusion and Low-Light Image Enhancement
• Revealing the Evolution of Order in Materials Microstructures Using Multi-Modal Computer Vision
• DeepIPCv2: LiDAR-powered Robust Environmental Perception and Navigational Control for Autonomous Vehicle