Point-Voxel CNN for Efficient 3D Deep Learning

Abstract

We present Point-Voxel CNN (PVCNN) for efficient, fast 3D deep learning. Previous work processes 3D data using either voxel-based or point-based NN models. However, both approaches are computationally inefficient. The computation cost and memory footprints of the voxel-based models grow cubically with the input resolution, making it memory-prohibitive to scale up the resolution. As for point-based networks, up to 80% of the time is wasted on structuring the sparse data which have rather poor memory locality, not on the actual feature extraction. In this paper, we propose PVCNN that represents the 3D input data in points to reduce the memory consumption, while performing the convolutions in voxels to reduce the irregular, sparse data access and improve the locality. Our PVCNN model is both memory and computation efficient. Evaluated on semantic and part segmentation datasets, it achieves much higher accuracy than the voxel-based baseline with 10x GPU memory reduction; it also outperforms the state-of-the-art point-based models with 7x measured speedup on average. Remarkably, the narrower version of PVCNN achieves 2x speedup over PointNet (an extremely efficient model) on part and scene segmentation benchmarks with much higher accuracy. We validate the general effectiveness of PVCNN on 3D object detection: by replacing the primitives in Frustrum PointNet with PVConv, it outperforms Frustrum PointNet++ by 2.4% mAP on average with 1.5x measured speedup and GPU memory reduction.

NeurIPS 2019. The first two authors contributed equally to this work. Project page: http://pvcnn.mit.edu/

References in corpus (7)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• Fixed Point Quantization of Deep Convolutional Networks
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks

Cited by in corpus (24)

• Comprehensive Review of Deep Learning-Based 3D Point Cloud Completion Processing and Analysis
• PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
• Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• FPS-Net: A Convolutional Fusion Network for Large-Scale LiDAR Point Cloud Segmentation
• Learning Part Boundaries from 3D Point Clouds
• InversionNet3D: Efficient and Scalable Learning for 3D Full Waveform Inversion
• FG-Net: Fast Large-Scale LiDAR Point Clouds Understanding Network Leveraging Correlated Feature Mining and Geometric-Aware Modelling
• Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots
• Two-and-a-half Order Score-based Model for Solving 3D Ill-posed Inverse Problems
• Machine-learning based methodologies for 3d x-ray measurement, characterization and optimization for buried structures in advanced ic packages
• Efficient Anatomical Labeling of Pulmonary Tree Structures via Deep Point-Graph Representation-based Implicit Fields
• Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data
• Grid-GCN for Fast and Scalable Point Cloud Learning
• ePointDA: An End-to-End Simulation-to-Real Domain Adaptation Framework for LiDAR Point Cloud Segmentation
• MS3D++: Ensemble of Experts for Multi-Source Unsupervised Domain Adaptation in 3D Object Detection
• Automatic Segmentation of Aircraft Dents in Point Clouds
• SynthmanticLiDAR: A Synthetic Dataset for Semantic Segmentation on LiDAR Imaging
• Lightweight integration of 3D features to improve 2D image segmentation
• Real-Time Spatio-Temporal LiDAR Point Cloud Compression
• AdvectiveNet: An Eulerian-Lagrangian Fluidic reservoir for Point Cloud Processing
• Analysis of memory consumption by neural networks based on hyperparameters
• Background-Aware 3D Point Cloud Segmentationwith Dynamic Point Feature Aggregation
• Car Drag Coefficient Prediction from 3D Point Clouds Using a Slice-Based Surrogate Model