Unsupervised Discovery of Object Radiance Fields
arXiv:2107.07905
Abstract
We study the problem of inferring an object-centric scene representation from a single image, aiming to derive a representation that explains the image formation process, captures the scene's 3D nature, and is learned without supervision. Most existing methods on scene decomposition lack one or more of these characteristics, due to the fundamental challenge in integrating the complex 3D-to-2D image formation process into powerful inference schemes like deep networks. In this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating recent progresses in neural 3D scene representations and rendering with deep inference networks for unsupervised 3D scene decomposition. Trained on multi-view RGB images without annotations, uORF learns to decompose complex scenes with diverse, textured background from a single image. We show that uORF enables novel tasks, such as scene segmentation and editing in 3D, and it performs well on these tasks and on novel view synthesis on three datasets.
arXiv commentICLR 2022. Project page: https://kovenyu.com/uorf/
References in corpus (14)
- Which Training Methods for GANs do actually Converge?
- DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
- Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
- Multi-Object Representation Learning with Iterative Variational Inference
- MONet: Unsupervised Scene Decomposition and Representation
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Differentiable Rendering: A Survey
- Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
- SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
- Object-Centric Neural Scene Rendering
- Toward unsupervised, multi-object discovery in large-scale image collections
- ShaRF: Shape-conditioned Radiance Fields from a Single View
- Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
- NeRF-VAE: A Geometry Aware 3D Scene Generative Model