Deep Depth Completion of a Single RGB-D Image

Abstract

The goal of our work is to complete the depth channel of an RGB-D image. Commodity-grade depth cameras often fail to sense depth for shiny, bright, transparent, and distant surfaces. To address this problem, we train a deep network that takes an RGB image as input and predicts dense surface normals and occlusion boundaries. Those predictions are then combined with raw depth observations provided by the RGB-D camera to solve for depths for all pixels, including those missing in the original observation. This method was chosen over others (e.g., inpainting depths directly) as the result of extensive experiments with a new depth completion benchmark dataset, where holes are filled in training data through the rendering of surface reconstructions created from multiview RGB-D scans. Experiments with different network inputs, depth representations, loss functions, optimization methods, inpainting methods, and deep depth estimation networks show that our proposed approach provides better depth completions than these alternatives.

Accepted by CVPR2018 (Spotlight). Project webpage: http://deepcompletion.cs.princeton.edu/ This version includes supplementary materials which provide more implementation details, quantitative evaluation, and qualitative results. Due to file size limit, please check project website for high-res paper

References in corpus (6)

• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image
• Semantic Scene Completion from a Single Depth Image
• Designing Deep Networks for Surface Normal Estimation

Cited by in corpus (12)

• Learning Guided Convolutional Network for Depth Completion
• Stereo Correspondence and Reconstruction of Endoscopic Data Challenge
• Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera
• 3D Photography using Context-aware Layered Depth Inpainting
• Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation
• CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion
• Unsupervised Enhancement of Real-World Depth Images Using Tri-Cycle GAN
• FrameNet: Learning Local Canonical Frames of 3D Surfaces from a Single RGB Image
• Normal Assisted Stereo Depth Estimation
• MG-SAGC: A multiscale graph and its self-adaptive graph convolution network for 3D point clouds
• Fast Generation of High Fidelity RGB-D Images by Deep-Learning with Adaptive Convolution
• Brain Tumor Segmentation on MRI with Missing Modalities