3D Photography using Context-aware Layered Depth Inpainting
arXiv:2004.04727
Abstract
We propose a method for converting a single RGB-D input image into a 3D photo - a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts compared with the state of the arts.
arXiv commentCVPR 2020. Project page: https://shihmengli.github.io/3D-Photo-Inpainting/ Code: https://github.com/vt-vl-lab/3d-photo-inpainting Demo: https://colab.research.google.com/drive/1706ToQrkIZshRSJSHvZ1RuCiM__YX3Bz
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Spectral Normalization for Generative Adversarial Networks
- Instance Normalization: The Missing Ingredient for Fast Stylization
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Unsupervised Learning of Depth and Ego-Motion from Video
- Generative Image Inpainting with Contextual Attention
- Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
- DeepStereo: Learning to Predict New Views from the World's Imagery
- Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
- Foreground-aware Image Inpainting
- Deep Depth Completion of a Single RGB-D Image
- Geometry-aware Deep Network for Single-Image Novel View Synthesis
- Pushing the Boundaries of View Extrapolation with Multiplane Images
- Contextual-based Image Inpainting: Infer, Match, and Translate
- Extreme View Synthesis
- Layer-structured 3D Scene Inference via View Synthesis