Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

Abstract

We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry. To tackle this problem, we create a new scan-to-CAD alignment dataset based on 1506 ScanNet scans with 97607 annotated keypoint pairs between 14225 CAD models from ShapeNet and their counterpart objects in the scans. Our method selects a set of representative keypoints in a 3D scan for which we find correspondences to the CAD geometry. To this end, we design a novel 3D CNN architecture that learns a joint embedding between real and synthetic objects, and from this predicts a correspondence heatmap. Based on these correspondence heatmaps, we formulate a variational energy minimization that aligns a given set of CAD models to the reconstruction. We evaluate our approach on our newly introduced Scan2CAD benchmark where we outperform both handcrafted feature descriptor as well as state-of-the-art CNN based methods by 21.39%.

Video: https://youtu.be/PiHSYpgLTfA

References in corpus (9)

• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
• ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
• Semantic Scene Completion from a Single Depth Image
• Fusion++: Volumetric Object-Level SLAM
• Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image

Cited by in corpus (10)

• Deep-SLAM++: Object-level RGBD SLAM based on class-specific deep shape priors
• Fast 3D Indoor Scene Synthesis with Discrete and Exact Layout Pattern Extraction
• Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
• RevealNet: Seeing Behind Objects in RGB-D Scans
• Canonical Voting: Towards Robust Oriented Bounding Box Detection in 3D Scenes
• Towards Panoptic 3D Parsing for Single Image in the Wild
• Interactive Annotation of 3D Object Geometry using 2D Scribbles
• MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos
• $360^o$ Surface Regression with a Hyper-Sphere Loss