Deep Continuous Fusion for Multi-Sensor 3D Object Detection

Abstract

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.

ECCV 2018

References in corpus (2)

• Semi-Supervised Classification with Graph Convolutional Networks
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

Cited by in corpus (54)

• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
• End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds
• Towards Robust LiDAR-based Perception in Autonomous Driving: General Black-box Adversarial Sensor Attack and Countermeasures
• Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Loss
• Self-Supervised Few-Shot Learning on Point Clouds
• RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Driving
• CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection
• Stereo R-CNN based 3D Object Detection for Autonomous Driving
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• 3D Point Cloud Processing and Learning for Autonomous Driving
• P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
• 3D IoU-Net: IoU Guided 3D Object Detector for Point Clouds
• Pillar-based Object Detection for Autonomous Driving
• TANet: Robust 3D Object Detection from Point Clouds with Triple Attention
• Exploring Data Augmentation for Multi-Modality 3D Object Detection
• Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review
• Multi-View Adaptive Fusion Network for 3D Object Detection
• MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships
• SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud
• SIENet: Spatial Information Enhancement Network for 3D Object Detection from Point Cloud
• Associate-3Ddet: Perceptual-to-Conceptual Association for 3D Point Cloud Object Detection
• V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation
• Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction
• Train in Germany, Test in The USA: Making 3D Object Detectors Generalize
• RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects
• An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
• MAFF-Net: Filter False Positive for 3D Vehicle Detection with Multi-modal Adaptive Feature Fusion
• SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds
• End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection
• PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Module
• PnPNet: End-to-End Perception and Prediction with Tracking in the Loop
• Multi-Task Multi-Sensor Fusion for 3D Object Detection
• MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
• Semi-Supervised Image Deraining using Gaussian Processes
• GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision
• Self-Supervised Pillar Motion Learning for Autonomous Driving
• Review: deep learning on 3D point clouds
• Cross-Modality 3D Object Detection
• Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting
• Learning Joint 2D-3D Representations for Depth Completion
• Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models
• Multi-view PointNet for 3D Scene Understanding
• BiFNet: Bidirectional Fusion Network for Road Segmentation
• InfoFocus: 3D Object Detection for Autonomous Driving with Dynamic Information Modeling
• LiDAR R-CNN: An Efficient and Universal 3D Object Detector
• Large-scale 3D point cloud representations via graph inception networks with applications to autonomous driving
• Multi-scale Receptive Fields Graph Attention Network for Point Cloud Classification
• Convolutional Recurrent Network for Road Boundary Extraction
• R-AGNO-RPN: A LIDAR-Camera Region Deep Network for Resolution-Agnostic Detection
• RoIFusion: 3D Object Detection from LiDAR and Vision
• Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving