SPLATNet: Sparse Lattice Networks for Point Cloud Processing

Abstract

We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice. Naively applying convolutions on this lattice scales poorly, both in terms of memory and computational cost, as the size of the lattice increases. Instead, our network uses sparse bilateral convolutional layers as building blocks. These layers maintain efficiency by using indexing structures to apply convolutions only on occupied parts of the lattice, and allow flexible specifications of the lattice structure enabling hierarchical and spatially-aware feature learning, as well as joint 2D-3D reasoning. Both point-based and image-based representations can be easily incorporated in a network with such layers and the resulting model can be trained in an end-to-end manner. We present results on 3D segmentation tasks where our approach outperforms existing state-of-the-art techniques.

arXiv commentCamera-ready, accepted to CVPR 2018 (oral); project website: http://vis-www.cs.umass.edu/splatnet/

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering
• Deep Convolutional Networks on Graph-Structured Data
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• FusionNet: 3D Object Classification Using Multiple Data Representations
• SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation
• GIFT: A Real-time and Scalable 3D Shape Search Engine
• OctNet: Learning Deep 3D Representations at High Resolutions
• OctNetFusion: Learning Depth Fusion from Data

Cited by in corpus (20)

• RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds
• PointCNN: Convolution On $\mathcal{X}$-Transformed Points
• SPNets: Differentiable Fluid Dynamics for Deep Neural Networks
• Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud
• Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation
• PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention
• Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN
• Deformable Filter Convolution for Point Cloud Reasoning
• Iterative Distance-Aware Similarity Matrix Convolution with Mutual-Supervised Point Elimination for Efficient Point Cloud Registration
• CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision
• AdvectiveNet: An Eulerian-Lagrangian Fluidic reservoir for Point Cloud Processing
• What and Where: Modeling Skeletons from Semantic and Spatial Perspectives for Action Recognition
• Discrete Point Flow Networks for Efficient Point Cloud Generation
• Multi Voxel-Point Neurons Convolution (MVPConv) for Fast and Accurate 3D Deep Learning
• A New Framework for Registration of Semantic Point Clouds from Stereo and RGB-D Cameras
• Going Deeper with Lean Point Networks
• Point Cloud Colorization Based on Densely Annotated 3D Shape Dataset
• Exploring Deep 3D Spatial Encodings for Large-Scale 3D Scene Understanding
• Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving
• Adversarial Self-Supervised Scene Flow Estimation