ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans

Abstract

We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels. The key contribution of our method is its ability to handle large scenes with varying spatial extent, managing the cubic growth in data size as scene size increases. To this end, we devise a fully-convolutional generative 3D CNN model whose filter kernels are invariant to the overall scene size. The model can be trained on scene subvolumes but deployed on arbitrarily large scenes at test time. In addition, we propose a coarse-to-fine inference strategy in order to produce high-resolution output while also leveraging large input context sizes. In an extensive series of experiments, we carefully evaluate different model design choices, considering both deterministic and probabilistic models for completion and semantic inference. Our results show that we outperform other methods not only in the size of the environments handled and processing efficiency, but also with regard to completion quality and semantic segmentation performance by a significant margin.

arXiv commentVideo: https://youtu.be/5s5s8iH0NF8

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• Parallel Multiscale Autoregressive Density Estimation
• High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference
• OctNetFusion: Learning Depth Fusion from Data

Cited by in corpus (14)

• Spatial Transformer for 3D Point Clouds
• Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
• 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
• Real-time Progressive 3D Semantic Segmentation for Indoor Scene
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
• View-volume Network for Semantic Scene Completion from a Single Depth Image
• ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image
• RevealNet: Seeing Behind Objects in RGB-D Scans
• 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation
• DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors
• Scan2Mesh: From Unstructured Range Scans to 3D Meshes
• End-to-end Generative Floor-plan and Layout with Attributes and Relation Graph
• 3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video
• Towards Part-Based Understanding of RGB-D Scans