Deep Continuous Fusion for Multi-Sensor 3D Object Detection

Abstract

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.

ECCV 2018

References in corpus (7)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• Geometric deep learning: going beyond Euclidean data
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• Learning shape correspondence with anisotropic convolutional neural networks

Cited by in corpus (123)

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
• 3D Object Detection for Autonomous Driving: A Survey
• Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
• Dynamic Graph CNN for Learning on Point Clouds
• Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks
• PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
• PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud
• Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
• From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network
• End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds
• Towards Robust LiDAR-based Perception in Autonomous Driving: General Black-box Adversarial Sensor Attack and Countermeasures
• Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving
• From SLAM to Situational Awareness: Challenges and Survey
• Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Loss
• RangeRCNN: Towards Fast and Accurate 3D Object Detection with Range Image Representation
• Object DGCNN: 3D Object Detection using Dynamic Graphs
• Deep Learning for 3D Point Clouds: A Survey
• Self-Supervised Few-Shot Learning on Point Clouds
• What Makes Multi-modal Learning Better than Single (Provably)
• RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Driving
• CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection
• SemanticVoxels: Sequential Fusion for 3D Pedestrian Detection using LiDAR Point Cloud and Semantic Segmentation
• Stereo R-CNN based 3D Object Detection for Autonomous Driving
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
• 3D Point Cloud Processing and Learning for Autonomous Driving
• A Survey of Simultaneous Localization and Mapping with an Envision in 6G Wireless Networks
• 3D IoU-Net: IoU Guided 3D Object Detector for Point Clouds
• Pillar-based Object Detection for Autonomous Driving
• PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection
• TANet: Robust 3D Object Detection from Point Clouds with Triple Attention
• Deep Learning for 3D Point Cloud Understanding: A Survey
• Exploring Data Augmentation for Multi-Modality 3D Object Detection
• Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review
• DSGN: Deep Stereo Geometry Network for 3D Object Detection
• Multi-View Adaptive Fusion Network for 3D Object Detection
• HVNet: Hybrid Voxel Network for LiDAR Based 3D Object Detection
• MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships
• Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots
• Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes
• F-Cooper: Feature based Cooperative Perception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds
• SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud
• Multi-Modal 3D Object Detection in Autonomous Driving: a Survey
• SIENet: Spatial Information Enhancement Network for 3D Object Detection from Point Cloud
• Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving
• Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving
• IoU Loss for 2D/3D Object Detection
• Associate-3Ddet: Perceptual-to-Conceptual Association for 3D Point Cloud Object Detection
• On Single Source Robustness in Deep Fusion Models
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation
• Deformable Filter Convolution for Point Cloud Reasoning
• V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction
• FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras
• Train in Germany, Test in The USA: Making 3D Object Detectors Generalize
• Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction
• PASS3D: Precise and Accelerated Semantic Segmentation for 3D Point Cloud
• ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving
• Improving 3D Object Detection through Progressive Population Based Augmentation
• LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion
• An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
• Improving 3D Object Detection with Channel-wise Transformer
• RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects
• SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generation
• Security Analysis of Camera-LiDAR Fusion Against Black-Box Attacks on Autonomous Vehicles
• Multiview Detection with Feature Perspective Transformation
• MAFF-Net: Filter False Positive for 3D Vehicle Detection with Multi-modal Adaptive Feature Fusion
• SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds
• End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection
• PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Module
• Learning to Communicate and Correct Pose Errors
• MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps
• Multi-Sensor 3D Object Box Refinement for Autonomous Driving
• PnPNet: End-to-End Perception and Prediction with Tracking in the Loop
• Multi-Task Multi-Sensor Fusion for 3D Object Detection
• MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
• Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation
• 3DRM:Pair-wise relation module for 3D object detection
• From Voxel to Point: IoU-guided 3D Object Detection for Point Cloud with Voxel-to-Point Decoder
• GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision
• Semi-Supervised Image Deraining using Gaussian Processes
• Self-Supervised Pillar Motion Learning for Autonomous Driving
• Multi-Modality Task Cascade for 3D Object Detection
• Voxel Transformer for 3D Object Detection
• Review: deep learning on 3D point clouds
• CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation
• Cross-Modality 3D Object Detection
• Monocular 3D Object Detection: An Extrinsic Parameter Free Approach
• A Review on Object Pose Recovery: from 3D Bounding Box Detectors to Full 6D Pose Estimators
• Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting
• Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models
• Multi-view PointNet for 3D Scene Understanding
• Investigating Attention Mechanism in 3D Point Cloud Object Detection
• Learning Joint 2D-3D Representations for Depth Completion
• MBDF-Net: Multi-Branch Deep Fusion Network for 3D Object Detection
• Stereo RGB and Deeper LIDAR Based Network for 3D Object Detection
• LiDAR R-CNN: An Efficient and Universal 3D Object Detector
• BiFNet: Bidirectional Fusion Network for Road Segmentation
• InfoFocus: 3D Object Detection for Autonomous Driving with Dynamic Information Modeling
• MLOD: A multi-view 3D object detection based on robust feature fusion method
• Monocular Multi-Layer Layout Estimation for Warehouse Racks
• Vision-based Lifting of 2D Object Detections for Automated Driving
• Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles
• AutoLay: Benchmarking amodal layout estimation for autonomous driving
• Unsupervised Object Detection with LiDAR Clues
• Multi-scale Receptive Fields Graph Attention Network for Point Cloud Classification
• Large-scale 3D point cloud representations via graph inception networks with applications to autonomous driving
• Dynamic Convolution for 3D Point Cloud Instance Segmentation
• R-AGNO-RPN: A LIDAR-Camera Region Deep Network for Resolution-Agnostic Detection
• Adaptive and Azimuth-Aware Fusion Network of Multimodal Local Features for 3D Object Detection
• Convolutional Recurrent Network for Road Boundary Extraction
• Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving
• Boundary-Aware Dense Feature Indicator for Single-Stage 3D Object Detection from Point Clouds
• DV-Det: Efficient 3D Point Cloud Object Detection with Dynamic Voxelization
• RoIFusion: 3D Object Detection from LiDAR and Vision
• Cascaded Refinement Network for Point Cloud Completion with Self-supervision
• Context-Aware Dynamic Feature Extraction for 3D Object Detection in Point Clouds
• Stereo Frustums: A Siamese Pipeline for 3D Object Detection