3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Abstract

Matching local geometric features on real-world depth images is a challenging task due to the noisy, low-resolution, and incomplete nature of 3D scan data. These difficulties limit the performance of current state-of-art methods, which are typically based on histograms over geometric properties. In this paper, we present 3DMatch, a data-driven model that learns a local volumetric patch descriptor for establishing correspondences between partial 3D data. To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions. Experiments show that our descriptor is not only able to match local geometry in new scenes for reconstruction, but also generalize to different tasks and spatial scales (e.g. instance-level object model alignment for the Amazon Picking Challenge, and mesh surface correspondence). Results show that 3DMatch consistently outperforms other state-of-the-art approaches by a significant margin. Code, data, benchmarks, and pre-trained models are available online at http://3dmatch.cs.princeton.edu

To appear at the Conference on Computer Vision and Pattern Recognition (CVPR) 2017. Project webpage: http://3dmatch.cs.princeton.edu

References in corpus (2)

• Learning to See by Moving
• Unsupervised Learning of Spatiotemporally Coherent Metrics

Cited by in corpus (33)

• RPM-Net: Robust Point Matching using Learned Features
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• PPFNet: Global Context Aware Local Features for Robust 3D Point Matching
• The Perfect Match: 3D Point Cloud Matching with Smoothed Densities
• D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features
• 3D-A-Nets: 3D Deep Dense Descriptor for Volumetric Shapes with Adversarial Networks
• Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
• DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
• You Only Hypothesize Once: Point Cloud Registration with Rotation-equivariant Descriptors
• USIP: Unsupervised Stable Interest Point Detection from 3D Point Clouds
• Octree guided CNN with Spherical Kernels for 3D Point Clouds
• SOE-Net: A Self-Attention and Orientation Encoding Network for Point Cloud based Place Recognition
• Fast 3D Indoor Scene Synthesis with Discrete and Exact Layout Pattern Extraction
• DeepMapping: Unsupervised Map Estimation From Multiple Point Clouds
• End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds
• IMFNet: Interpretable Multimodal Fusion for Point Cloud Registration
• DFC: Deep Feature Consistency for Robust Point Cloud Registration
• Unsupervised Learning of Intrinsic Structural Representation Points
• Learning Local Shape Descriptors from Part Correspondences With Multi-view Convolutional Networks
• StablePose: Learning 6D Object Poses from Geometrically Stable Patches
• Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
• Learning to Optimize Non-Rigid Tracking
• Scan2Mesh: From Unstructured Range Scans to 3D Meshes
• DUGMA: Dynamic Uncertainty-Based Gaussian Mixture Alignment
• Floorplan-Jigsaw: Jointly Estimating Scene Layout and Aligning Partial Scans
• What Stops Learning-based 3D Registration from Working in the Real World?
• A Robust Loss for Point Cloud Registration
• Equivariant Point Network for 3D Point Cloud Analysis
• Registration Loss Learning for Deep Probabilistic Point Set Registration
• InSphereNet: a Concise Representation and Classification Method for 3D Object
• Decoupling Features and Coordinates for Few-shot RGB Relocalization
• Torch-Points3D: A Modular Multi-Task Frameworkfor Reproducible Deep Learning on 3D Point Clouds