Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review

Abstract

Driven by deep learning techniques, perception technology in autonomous driving has developed rapidly in recent years, enabling vehicles to accurately detect and interpret surrounding environment for safe and efficient navigation. To achieve accurate and robust perception capabilities, autonomous vehicles are often equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these fused sensors, radars and cameras enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. This review aims to provide a comprehensive guideline for radar-camera fusion, particularly concentrating on perception tasks related to object detection and semantic segmentation.Based on the principles of the radar and camera sensors, we delve into the data processing process and representations, followed by an in-depth analysis and summary of radar-camera fusion datasets. In the review of methodologies in radar-camera fusion, we address interrogative questions, including "why to fuse", "what to fuse", "where to fuse", "when to fuse", and "how to fuse", subsequently discussing various challenges and potential research directions within this domain. To ease the retrieval and comparison of datasets and fusion methods, we also provide an interactive website: https://radar-camera-fusion.github.io.

arXiv commentAccepted by IEEE Transactions on Intelligent Vehicles (T-IV)

References in corpus (50)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• YOLOv4: Optimal Speed and Accuracy of Object Detection
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Methods for Interpreting and Understanding Deep Neural Networks
• Event-based Vision: A Survey
• PCT: Point cloud transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• The ApolloScape Open Dataset for Autonomous Driving and its Application
• Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
• Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation
• CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection
• Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks
• Multimodal End-to-End Autonomous Driving
• Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework
• CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks
• Self-Supervised Model Adaptation for Multimodal Semantic Segmentation
• RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization
• RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehicles
• RAMP-CNN: A Novel Neural Network for Enhanced Automotive Radar Object Recognition
• Simultaneous Traffic Sign Detection and Boundary Estimation using Convolutional Neural Network
• CNN based Road User Detection using the 3D Radar Cube
• 2D Car Detection in Radar Data with PointNets
• Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar
• TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving
• Towards Domain-Independent and Real-Time Gesture Recognition Using mmWave Signal
• HRFormer: High-Resolution Transformer for Dense Prediction
• Automotive Radar Interference Mitigation Using Adaptive Noise Canceller
• LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion
• Experiments with mmWave Automotive Radar Test-bed
• Deep Instance Segmentation with Automotive Radar Detection Points
• How much real data do we actually need: Analyzing object detection performance using synthetic and real data
• Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
• Radar Voxel Fusion for 3D Object Detection
• Probabilistic End-to-End Vehicle Navigation in Complex Dynamic Environments with Multimodal Sensor Fusion
• Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention
• A Comparative Survey of Deep Active Learning
• RadarGNN: Transformation Invariant Graph Neural Network for Radar-based Perception
• Gaussian Radar Transformer for Semantic Segmentation in Noisy Radar Data
• Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles
• Exploit Camera Raw Data for Video Super-Resolution via Hidden Markov Model Inference
• Improving Multi-Modal Learning with Uni-Modal Teachers
• Radar Image Reconstruction from Raw ADC Data using Parametric Variational Autoencoder with Domain Adaptation
• Improved Multi-Scale Grid Rendering of Point Clouds for Radar Object Detection Networks
• RadSegNet: A Reliable Approach to Radar Camera Fusion
• aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception

Cited by in corpus (10)

• LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion
• Exploring Radar Data Representations in Autonomous Driving: A Comprehensive Review
• Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervision
• RadarCNN: Learning-based Indoor Object Classification from IQ Imaging Radar Data
• On-the-Fly Interrogation of Mobile Passive Sensors from the Fusion of Optical and Radar Data
• A Low-Complexity PFA-Based Autofocus Algorithm for Automotive SAR
• A Resource Efficient Fusion Network for Object Detection in Bird's-Eye View using Camera and Raw Radar Data
• Advancing Autonomous Driving: DepthSense with Radar and Spatial Attention
• REFNet++: Multi-Task Efficient Fusion of Camera and Radar Sensor Data in Bird's-Eye Polar View
• Reflection-Aware Reasoning for Non-Line-of-Sight Pedestrian Localization