3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection

Abstract

In this paper, we propose a new deep architecture for fusing camera and LiDAR sensors for 3D object detection. Because the camera and LiDAR sensor signals have different characteristics and distributions, fusing these two modalities is expected to improve both the accuracy and robustness of 3D object detection. One of the challenges presented by the fusion of cameras and LiDAR is that the spatial feature maps obtained from each modality are represented by significantly different views in the camera and world coordinates; hence, it is not an easy task to combine two heterogeneous feature maps without loss of information. To address this problem, we propose a method called 3D-CVF that combines the camera and LiDAR features using the cross-view spatial feature fusion strategy. First, the method employs auto-calibrated projection, to transform the 2D camera features to a smooth spatial feature map with the highest correspondence to the LiDAR features in the bird's eye view (BEV) domain. Then, a gated feature fusion network is applied to use the spatial attention maps to mix the camera and LiDAR features appropriately according to the region. Next, camera-LiDAR feature fusion is also achieved in the subsequent proposal refinement stage. The camera feature is used from the 2D camera-view domain via 3D RoI grid pooling and fused with the BEV feature for proposal refinement. Our evaluations, conducted on the KITTI and nuScenes 3D object detection datasets demonstrate that the camera-LiDAR fusion offers significant performance gain over single modality and that the proposed 3D-CVF achieves state-of-the-art performance in the KITTI benchmark.

References in corpus (1)

• SSD: Single Shot MultiBox Detector

Cited by in corpus (15)

• Adaptive Context-Aware Multi-Modal Network for Depth Completion
• Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds
• Fully Sparse Fusion for 3D Object Detection
• VoxelNextFusion: A Simple, Unified and Effective Voxel Fusion Framework for Multi-Modal 3D Object Detection
• PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection
• Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
• Multi-Modal 3D Object Detection in Autonomous Driving: a Survey
• MOISST: Multimodal Optimization of Implicit Scene for SpatioTemporal calibration
• ClusterFusion: Leveraging Radar Spatial Features for Radar-Camera 3D Object Detection in Autonomous Vehicles
• Paint and Distill: Boosting 3D Object Detection with Semantic Passing Network
• Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving
• Robust LiDAR-Camera Calibration with 2D Gaussian Splatting
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception
• CAR -- Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles