SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
arXiv:2006.14660
Abstract
We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion. Our self-supervised approach learns to jointly inpaint geometry and color by correlating an incomplete RGB-D scan with a more complete version of that scan. Notably, rather than relying on 3D reconstruction losses to inform our 3D geometry and color reconstruction, we propose adversarial and perceptual losses operating on 2D renderings in order to achieve high-resolution, high-quality colored reconstructions of scenes. This exploits the high-resolution, self-consistent signal from individual raw RGB-D frames, in contrast to fused 3D reconstructions of the frames which exhibit inconsistencies from view-dependent effects, such as color balancing or pose inconsistencies. Thus, by informing our 3D scene generation directly through 2D signal, we produce high-quality colored reconstructions of 3D scenes, outperforming state of the art on both synthetic and real data.
arXiv commentVideo: https://youtu.be/1cj962m9zqo
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- Im2Avatar: Colorful 3D Reconstruction from a Single Image
- Convolutional Occupancy Networks
- Adversarial Texture Optimization from RGB-D Scans