Peeking Behind Objects: Layered Depth Prediction from a Single Image

Abstract

While conventional depth estimation can infer the geometry of a scene from a single RGB image, it fails to estimate scene regions that are occluded by foreground objects. This limits the use of depth prediction in augmented and virtual reality applications, that aim at scene exploration by synthesizing the scene from a different vantage point, or at diminished reality. To address this issue, we shift the focus from conventional depth map prediction to the regression of a specific data representation called Layered Depth Image (LDI), which contains information about the occluded regions in the reference frame and can fill in occlusion gaps in case of small view changes. We propose a novel approach based on Convolutional Neural Networks (CNNs) to jointly predict depth maps and foreground separation masks used to condition Generative Adversarial Networks (GANs) for hallucinating plausible color and depths in the initially occluded areas. We demonstrate the effectiveness of our approach for novel scene view synthesis from a single image.

References in corpus (13)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
• Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation
• Depth from a Single Image by Harmonizing Overcomplete Local Network Predictions
• View Synthesis by Appearance Flow
• Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks
• Layer-structured 3D Scene Inference via View Synthesis

Cited by in corpus (5)

• Image inpainting: A review
• Image Amodal Completion: A Survey
• RGB-D Image Inpainting Using Generative Adversarial Network with a Late Fusion Approach
• Semantic View Synthesis
• Boundary-induced and scene-aggregated network for monocular depth prediction