Object-Centric Neural Scene Rendering

Abstract

We present a method for composing photorealistic scenes from captured images of objects. Our work builds upon neural radiance fields (NeRFs), which implicitly model the volumetric density and directionally-emitted radiance of a scene. While NeRFs synthesize realistic pictures, they only model static scenes and are closely tied to specific imaging conditions. This property makes NeRFs hard to generalize to new scenarios, including new lighting or new arrangements of objects. Instead of learning a scene radiance field as a NeRF does, we propose to learn object-centric neural scattering functions (OSFs), a representation that models per-object light transport implicitly using a lighting- and view-dependent neural network. This enables rendering scenes even when objects or lights move, without retraining. Combined with a volumetric path tracing procedure, our framework is capable of rendering both intra- and inter-object light transport effects including occlusions, specularities, shadows, and indirect illumination. We evaluate our approach on scene composition and show that it generalizes to novel illumination conditions, producing photorealistic, physically accurate renderings of multi-object scenes.

Summary Video: https://youtu.be/NtR7xgxSL1U Project Webpage: https://shellguo.com/osf

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• Single Image Portrait Relighting
• On the Spectral Bias of Neural Networks
• Neural Reflectance Fields for Appearance Acquisition
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• DeepStereo: Learning to Predict New Views from the World's Imagery
• LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering
• Neural Sparse Voxel Fields

Cited by in corpus (10)

• Neural Volume Rendering: NeRF And Beyond
• Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
• Unsupervised Discovery of Object Radiance Fields
• Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation
• MARF: The Medial Atom Ray Field Object Representation
• Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering
• Neural Fields in Visual Computing and Beyond
• Learning to Stylize Novel Views
• NeRFReN: Neural Radiance Fields with Reflections
• VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field