VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection

Abstract

Accurate detection of objects in 3D point clouds is a central problem in many applications, such as autonomous navigation, housekeeping robots, and augmented/virtual reality. To interface a highly sparse LiDAR point cloud with a region proposal network (RPN), most existing efforts have focused on hand-crafted feature representations, for example, a bird's eye view projection. In this work, we remove the need of manual feature engineering for 3D point clouds and propose VoxelNet, a generic 3D detection network that unifies feature extraction and bounding box prediction into a single stage, end-to-end trainable deep network. Specifically, VoxelNet divides a point cloud into equally spaced 3D voxels and transforms a group of points within each voxel into a unified feature representation through the newly introduced voxel feature encoding (VFE) layer. In this way, the point cloud is encoded as a descriptive volumetric representation, which is then connected to a RPN to generate detections. Experiments on the KITTI car detection benchmark show that VoxelNet outperforms the state-of-the-art LiDAR based 3D detection methods by a large margin. Furthermore, our network learns an effective discriminative representation of objects with various geometries, leading to encouraging results in 3D detection of pedestrians and cyclists, based on only LiDAR.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• Rich feature hierarchies for accurate object detection and semantic segmentation
• YOLO9000: Better, Faster, Stronger
• Vehicle Detection from 3D Lidar Using Fully Convolutional Network
• Multi-View 3D Object Detection Network for Autonomous Driving
• Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
• 3D Fully Convolutional Network for Vehicle Detection in Point Cloud

Cited by in corpus (50)

• PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud
• Joint 3D Proposal Generation and Object Detection from View Aggregation
• PointSeg: Real-Time Semantic Segmentation Based on 3D LiDAR Point Cloud
• PointConv: Deep Convolutional Networks on 3D Point Clouds
• DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
• RGCNN: Regularized Graph CNN for Point Cloud Segmentation
• 3D Object Detection Using Scale Invariant and Feature Reweighting Networks
• Complex-YOLO: Real-time 3D Object Detection on Point Clouds
• Adversarial Attack and Defense on Point Sets
• AutoCast: Scalable Infrastructure-less Cooperative Perception for Distributed Collaborative Driving
• STD: Sparse-to-Dense 3D Object Detector for Point Cloud
• Stereo R-CNN based 3D Object Detection for Autonomous Driving
• LiDARTag: A Real-Time Fiducial Tag System for Point Clouds
• Analyzing Infrastructure LiDAR Placement with Realistic LiDAR Simulation Library
• Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud
• Efficient Bird Eye View Proposals for 3D Siamese Tracking
• Where Should We Place LiDARs on the Autonomous Vehicle? - An Optimal Design Approach
• LiDAR Spoofing Meets the New-Gen: Capability Improvements, Broken Assumptions, and New Attack Strategies
• In Defense of Classical Image Processing: Fast Depth Completion on the CPU
• Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions
• SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud
• Flexible Supervised Autonomy for Exploration in Subterranean Environments
• SIENet: Spatial Information Enhancement Network for 3D Object Detection from Point Cloud
• Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN
• SESS: Self-Ensembling Semi-Supervised 3D Object Detection
• Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds
• 6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
• SAWNet: A Spatially Aware Deep Neural Network for 3D Point Cloud Processing
• LMNet: Real-time Multiclass Object Detection on CPU using 3D LiDAR
• Multi-Sensor 3D Object Box Refinement for Autonomous Driving
• Points2Pix: 3D Point-Cloud to Image Translation using conditional Generative Adversarial Networks
• Holistic Parameteric Reconstruction of Building Models from Point Clouds
• PointIT: A Fast Tracking Framework Based on 3D Instance Segmentation
• Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector
• View Invariant Human Body Detection and Pose Estimation from Multiple Depth Sensors
• 3D Object Detection From LiDAR Data Using Distance Dependent Feature Extraction
• Class-specific Anchoring Proposal for 3D Object Recognition in LIDAR and RGB Images
• Deep Person Detection in 2D Range Data
• Discrete Rotation Equivariance for Point Cloud Recognition
• PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
• Graph Neural Networks for Image Understanding Based on Multiple Cues: Group Emotion Recognition and Event Recognition as Use Cases
• Fast and Modular Autonomy Software for Autonomous Racing Vehicles
• FotonNet: A HW-Efficient Object Detection System Using 3D-Depth Segmentation and 2D-DNN Classifier
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
• Joint Spatial-Temporal Optimization for Stereo 3D Object Tracking
• To the Point: Efficient 3D Object Detection in the Range Image with Graph Convolution Kernels
• Not Only Look But Observe: Variational Observation Model of Scene-Level 3D Multi-Object Understanding for Probabilistic SLAM
• LiDAR point-cloud processing based on projection methods: a comparison
• Learning to Predict the 3D Layout of a Scene
• Anchor Distance for 3D Multi-Object Distance Estimation from 2D Single Shot