Semantic Scene Completion from a Single Depth Image

Abstract

This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (SSCNet), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and semantic labels for all voxels in the camera view frustum. Our network uses a dilation-based 3D context module to efficiently expand the receptive field and enable 3D context learning. To train our network, we construct SUNCG - a manually created large-scale dataset of synthetic 3D scenes with dense volumetric annotations. Our experiments demonstrate that the joint model outperforms methods addressing each task in isolation and outperforms alternative approaches on the semantic scene completion task.

References in corpus (3)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• SceneNet: Understanding Real World Indoor Scenes With Synthetic Data
• Predicting Complete 3D Models of Indoor Scenes

Cited by in corpus (28)

• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Cross-view Semantic Segmentation for Sensing Surroundings
• SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
• Delta Descriptors: Change-Based Place Representation for Robust Visual Localization
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
• ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• OctNetFusion: Learning Depth Fusion from Data
• Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
• Deep Depth Completion of a Single RGB-D Image
• Semantic Scene Completion Combining Colour and Depth: preliminary experiments
• 3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-scale 3D Point Clouds
• Pointwise Convolutional Neural Networks
• Learning High Dynamic Range from Outdoor Panoramas
• Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path
• Layer-structured 3D Scene Inference via View Synthesis
• Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery
• Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
• SurfConv: Bridging 3D and 2D Convolution for RGBD Images
• Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes
• Learning Single-Image Depth from Videos using Quality Assessment Networks
• Temporal and Physical Reasoning for Perception-Based Robotic Manipulation
• Learning to Separate Multiple Illuminants in a Single Image
• Unsupervised Deep Single-Image Intrinsic Decomposition using Illumination-Varying Image Sequences
• Support Relation Analysis for Objects in Multiple View RGB-D Images
• Incremental Scene Synthesis