Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Abstract

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of "what to fuse", "when to fuse", and "how to fuse" remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

Update copyright; Interactive platform online: https://boschresearch.github.io/multimodalperception/

References in corpus (50)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• End to End Learning for Self-Driving Cars
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Fully Convolutional Networks for Semantic Segmentation
• Conditional Random Fields as Recurrent Neural Networks
• Semi-Supervised Learning with Deep Generative Models
• A Review on Deep Learning Techniques Applied to Semantic Segmentation
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Learning to Reweight Examples for Robust Deep Learning
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Deep Continuous Fusion for Multi-Sensor 3D Object Detection
• Deep Parametric Continuous Convolutional Neural Networks
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• The EuroCity Persons Dataset: A Novel Benchmark for Object Detection
• Virtual Worlds as Proxy for Multi-Object Tracking Analysis
• IntentNet: Learning to Predict Intention from Raw Sensor Data
• HDNET: Exploiting HD Maps for 3D Object Detection
• Dimensionality-Driven Learning with Noisy Labels
• Deep Learning for Generic Object Detection: A Survey
• Do Convolutional Neural Networks Learn Class Hierarchy?
• Progressive LiDAR Adaptation for Road Detection
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Simultaneous Traffic Sign Detection and Boundary Estimation using Convolutional Neural Network
• Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection
• Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
• Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing
• Learning Factored Representations in a Deep Mixture of Experts
• Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving
• ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst
• StarNet: Targeted Computation for Object Detection in Point Clouds
• From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network
• Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments
• Deeply-Fused Nets
• Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation
• SynthCity: A large scale synthetic point cloud
• A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving
• Multi-modal Sensor Registration for Vehicle Perception via Deep Neural Networks
• Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Driving
• Blockout: Dynamic Model Selection for Hierarchical Deep Networks
• Fusing Bird View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection
• Multi-modal panoramic 3D outdoor datasets for place categorization

Cited by in corpus (89)

• A Survey of Autonomous Driving: Common Practices and Emerging Technologies
• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• 3D Object Detection for Autonomous Driving: A Survey
• CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection
• Cooperative Perception for 3D Object Detection in Driving Scenarios using Infrastructure Sensors
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• Multimodal End-to-End Autonomous Driving
• A Survey of End-to-End Driving: Architectures and Training Methods
• Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
• Canadian Adverse Driving Conditions Dataset
• RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization
• Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups
• DRL-VO: Learning to Navigate Through Crowded Dynamic Scenes Using Velocity Obstacles
• Polarization-driven Semantic Segmentation via Efficient Attention-bridged Fusion
• Review: Deep Learning in Electron Microscopy
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• RGB-D And Thermal Sensor Fusion: A Systematic Literature Review
• A Tutorial on 5G Positioning
• End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent
• Exploring Radar Data Representations in Autonomous Driving: A Comprehensive Review
• Behavioral Use Licensing for Responsible AI
• Motion Robust High-Speed Light-Weighted Object Detection With Event Camera
• An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving
• Enhancing Object Detection for Autonomous Driving by Optimizing Anchor Generation and Addressing Class Imbalance
• Dynamic Occupancy Grid Mapping with Recurrent Neural Networks
• Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving
• Universal image segmentation for optical identification of 2D materials
• Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion
• Bounding Box-Free Instance Segmentation Using Semi-Supervised Learning for Generating a City-Scale Vehicle Dataset
• FINE Samples for Learning with Noisy Labels
• Multi-scale Attentive Image De-raining Networks via Neural Architecture Search
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles
• Benchmarking Robustness of AI-Enabled Multi-sensor Fusion Systems: Challenges and Opportunities
• Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Driving
• I2F: A Unified Image-to-Feature Approach for Domain Adaptive Semantic Segmentation
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• Sharing to learn and learning to share; Fitting together Meta-Learning, Multi-Task Learning, and Transfer Learning: A meta review
• Organization of machine learning based product development as per ISO 26262 and ISO/PAS 21448
• Motion Estimation in Occupancy Grid Maps in Stationary Settings Using Recurrent Neural Networks
• CAE-LO: LiDAR Odometry Leveraging Fully Unsupervised Convolutional Auto-Encoder for Interest Point Detection and Feature Description
• Ad-datasets: a meta-collection of data sets for autonomous driving
• Radio astronomical images object detection and segmentation: A benchmark on deep learning methods
• Hidden Backdoor Attack against Semantic Segmentation Models
• Driving Datasets Literature Review
• Graph-Segmenter: Graph Transformer with Boundary-aware Attention for Semantic Segmentation
• Polar Parametrization for Vision-based Surround-View 3D Detection
• RELLIS-3D Dataset: Data, Benchmarks and Analysis
• ClusterFusion: Leveraging Radar Spatial Features for Radar-Camera 3D Object Detection in Autonomous Vehicles
• MultiTest: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systems
• On Single Source Robustness in Deep Fusion Models
• Key frames assisted hybrid encoding for photorealistic compressive video sensing
• Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Survey
• Transfer learning with generative models for object detection on limited datasets
• Semantic segmentation of longitudinal thermal images for identification of hot and cool spots in urban areas
• Training a Fast Object Detector for LiDAR Range Images Using Labeled Data from Sensors with Higher Resolution
• To Compute or not to Compute? Adaptive Smart Sensing in Resource-Constrained Edge Computing
• SYMOG: learning symmetric mixture of Gaussian modes for improved fixed-point quantization
• Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving
• DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments
• 3DRM:Pair-wise relation module for 3D object detection
• All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
• BoostRad: Enhancing Object Detection by Boosting Radar Reflections
• Composition and Application of Current Advanced Driving Assistance System: A Review
• Learning Autonomous Race Driving with Action Mapping Reinforcement Learning
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• DEAL: Difficulty-aware Active Learning for Semantic Segmentation
• Towards Sensor Data Abstraction of Autonomous Vehicle Perception Systems
• Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey and Experimental Study
• Deep learning-based astronomical multimodal data fusion: A comprehensive review
• Coarse-to-Fine Domain Adaptive Semantic Segmentation with Photometric Alignment and Category-Center Regularization
• Mineral segmentation using electron microscope images and spectral sampling through multimodal graph neural networks
• IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth
• Towards Better Performance and More Explainable Uncertainty for 3D Object Detection of Autonomous Vehicles
• Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation
• ISETAuto: Detecting vehicles with depth and radiance information
• Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation
• Radar-Camera Pixel Depth Association for Depth Completion
• Multimodal Knowledge Expansion
• A novel multimodal fusion network based on a joint coding model for lane line segmentation
• Memory based fusion for multi-modal deep learning
• Perspective, Survey and Trends: Public Driving Datasets and Toolsets for Autonomous Driving Virtual Test
• Parameter-Efficient Single Collaborative Branch for Recommendation
• Rethinking of Radar's Role: A Camera-Radar Dataset and Systematic Annotator via Coordinate Alignment
• Brno Urban Dataset: Winter Extention
• A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection