RevealNet: Seeing Behind Objects in RGB-D Scans
arXiv:1904.12012
Abstract
During 3D reconstruction, it is often the case that people cannot scan each individual object from all views, resulting in missing geometry in the captured scan. This missing geometry can be fundamentally limiting for many applications, e.g., a robot needs to know the unseen geometry to perform a precise grasp on an object. Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry. This will open up new possibilities for interactions with objects in a scene, for instance for virtual or robotic agents. We tackle this problem by introducing RevealNet, a new data-driven approach that jointly detects object instances and predicts their complete geometry. This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects, including hidden and unobserved object parts. RevealNet is an end-to-end 3D neural network architecture that leverages joint color and geometry feature learning. The fully-convolutional nature of our 3D network enables efficient inference of semantic instance completion for 3D scans at scale of large indoor environments in a single forward pass. We show that predicting complete object geometry improves both 3D detection and instance segmentation performance. We evaluate on both real and synthetic scan benchmark data for the new task, where we outperform state-of-the-art approaches by over 15 in [email protected] on ScanNet, and over 18 in [email protected] on SUNCG.
arXiv commentCVPR 2020
References in corpus (15)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
- ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
- Submanifold Sparse Convolutional Networks
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds
- MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation
- Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
- 3D-BEVIS: Bird's-Eye-View Instance Segmentation
- ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
- Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
- GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud
- Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
- 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
- PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things