PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes

Abstract

The rising availability of commercial $360^{\circ }$ cameras that democratize indoor scanning, has increased the interest for novel applications, such as interior space re-design. Diminished Reality (DR) fulfills the requirement of such applications, to remove existing objects in the scene, essentially translating this to a counterfactual inpainting task. While recent advances in data-driven inpainting have shown significant progress in generating realistic samples, they are not constrained to produce results with reality mapped structures. To preserve the `reality' in indoor (re-)planning applications, the scene's structure preservation is crucial. To ensure structure-aware counterfactual inpainting, we propose a model that initially predicts the structure of an indoor scene and then uses it to guide the reconstruction of an empty -- background only -- representation of the same scene. We train and compare against other state-of-the-art methods on a version of the Structured3D dataset modified for DR, showing superior results in both quantitative metrics and qualitative results, but more interestingly, our approach exhibits a much faster convergence rate. Code and models are available at https://vcl3d.github.io/PanoDR/ .

arXiv commentAccepted at CVPR, OmniCV Workshop. Code and models are available at https://vcl3d.github.io/PanoDR/

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Cited by in corpus (1)

• BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion