Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds

Abstract

Deep learning systems extensively use convolution operations to process input data. Though convolution is clearly defined for structured data such as 2D images or 3D volumes, this is not true for other data types such as sparse point clouds. Previous techniques have developed approximations to convolutions for restricted conditions. Unfortunately, their applicability is limited and cannot be used for general point clouds. We propose an efficient and effective method to learn convolutions for non-uniformly sampled point clouds, as they are obtained with modern acquisition techniques. Learning is enabled by four key novelties: first, representing the convolution kernel itself as a multilayer perceptron; second, phrasing convolution as a Monte Carlo integration problem, third, using this notion to combine information from multiple samplings at different levels; and fourth using Poisson disk sampling as a scalable means of hierarchical point cloud learning. The key idea across all these contributions is to guarantee adequate consideration of the underlying non-uniform sample distribution function from a Monte Carlo perspective. To make the proposed concepts applicable to real-world tasks, we furthermore propose an efficient implementation which significantly reduces the GPU memory required during the training process. By employing our method in hierarchical network architectures we can outperform most of the state-of-the-art networks on established point cloud segmentation, classification and normal estimation benchmarks. Furthermore, in contrast to most existing approaches, we also demonstrate the robustness of our method with respect to sampling variations, even when training with uniformly sampled data only. To support the direct application of these concepts, we provide a ready-to-use TensorFlow implementation of these layers at https://github.com/viscom-ulm/MCCNN

arXiv commentACM Transactions on Graphics (Proocedings of SIGGRAPH Asia 2018)

References in corpus (1)

• O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis

Cited by in corpus (23)

• PCT: Point cloud transformer
• Learning Semantic Segmentation of Large-Scale Point Clouds with Random Sampling
• DeepGCNs: Making GCNs Go as Deep as CNNs
• Deep Learning-based 3D Point Cloud Classification: A Systematic Survey and Outlook
• A review on deep learning techniques for 3D sensed data classification
• JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds
• Directionally Constrained Fully Convolutional Neural Network For Airborne Lidar Point Cloud Classification
• A Rotation-Invariant Framework for Deep Point Cloud Analysis
• A comprehensive overview of deep learning techniques for 3D point cloud classification and semantic segmentation
• LassoNet: Deep Lasso-Selection of 3D Point Clouds
• Density-Aware Convolutional Networks with Context Encoding for Airborne LiDAR Point Cloud Classification
• Advancements in Point Cloud-Based 3D Defect Detection and Classification for Industrial Systems: A Comprehensive Survey
• Learning Part Boundaries from 3D Point Clouds
• DeltaConv: Anisotropic Operators for Geometric Deep Learning on Point Clouds
• Continuous Conditional Random Field Convolution for Point Cloud Segmentation
• MKConv: Multidimensional Feature Representation for Point Cloud Analysis
• 3DMASC: Accessible, explainable 3D point clouds classification. Application to Bi-spectral Topo-bathymetric lidar data
• Cross-modal feature fusion for robust point cloud registration with ambiguous geometry
• BIMS-PU: Bi-Directional and Multi-Scale Point Cloud Upsampling
• Neural Field Convolutions by Repeated Differentiation
• Neural Gaussian Scale-Space Fields
• Local Latent Representation based on Geometric Convolution for Particle Data Feature Exploration
• Spectral Prefiltering of Neural Fields