Spatial Transformer for 3D Point Clouds

Abstract

Deep neural networks are widely used for understanding 3D point clouds. At each point convolution layer, features are computed from local neighborhoods of 3D points and combined for subsequent processing in order to extract semantic information. Existing methods adopt the same individual point neighborhoods throughout the network layers, defined by the same metric on the fixed input point coordinates. This common practice is easy to implement but not necessarily optimal. Ideally, local neighborhoods should be different at different layers, as more latent information is extracted at deeper layers. We propose a novel end-to-end approach to learn different non-rigid transformations of the input point cloud so that optimal local neighborhoods can be adopted at each layer. We propose both linear (affine) and non-linear (projective and deformable) spatial transformers for 3D point clouds. With spatial transformers on the ShapeNet part segmentation dataset, the network achieves higher accuracy for all categories, with 8\% gain on earphones and rockets in particular. Our method also outperforms the state-of-the-art on other point cloud tasks such as classification, detection, and semantic segmentation. Visualizations show that spatial transformers can learn features more efficiently by dynamically altering local neighborhoods according to the geometry and semantics of 3D shapes in spite of their within-category variations. Our code is publicly available at https://github.com/samaonline/spatial-transformer-for-3d-point-clouds.

arXiv commentTo appear in IEEE Transactions on PAMI, 2021

References in corpus (8)

• ShapeNet: An Information-Rich 3D Model Repository
• O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis
• Deep Parametric Continuous Convolutional Neural Networks
• ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
• Multiresolution Tree Networks for 3D Point Cloud Processing
• Learning and Matching Multi-View Descriptors for Registration of Point Clouds
• Fully-Convolutional Point Networks for Large-Scale Point Clouds
• Learning Sparse High Dimensional Filters: Image Filtering, Dense CRFs and Bilateral Neural Networks

Cited by in corpus (2)

• Segmentation of structural parts of rosebush plants with 3D point-based deep learning methods
• REGRAD: A Large-Scale Relational Grasp Dataset for Safe and Object-Specific Robotic Grasping in Clutter