Neural Reflectance Fields for Appearance Acquisition

Abstract

We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene using a fully-connected neural network. We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light. We demonstrate that neural reflectance fields can be estimated from images captured with a simple collocated camera-light setup, and accurately model the appearance of real-world scenes with complex geometry and reflectance. Once estimated, they can be used to render photo-realistic images under novel viewpoint and (non-collocated) lighting conditions and accurately reproduce challenging effects like specularities, shadows and occlusions. This allows us to perform high-quality view synthesis and relighting that is significantly better than previous methods. We also demonstrate that we can compose the estimated neural reflectance field of a real scene with traditional scene models and render them using standard Monte Carlo rendering engines. Our work thus enables a complete pipeline from high-quality and practical appearance acquisition to 3D scene composition and rendering.

References in corpus (8)

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• Single Image Portrait Relighting
• Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• Learning to Synthesize a 4D RGBD Light Field from a Single Image
• Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference
• Pushing the Boundaries of View Extrapolation with Multiplane Images

Cited by in corpus (20)

• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
• Non-line-of-Sight Imaging via Neural Transient Fields
• Object-Centric Neural Scene Rendering
• Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery
• Neural Volume Rendering: NeRF And Beyond
• Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing
• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
• Artemis: Articulated Neural Pets with Appearance and Motion synthesis
• NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
• TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
• VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization
• Neural Fields in Visual Computing and Beyond
• GS-ROR${}^2$: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstruction
• Template NeRF: Towards Modeling Dense Shape Correspondences from Category-Specific Object Images
• ReShader: View-Dependent Highlights for Single Image View-Synthesis
• Neural BRDFs: Representation and Operations
• Neural Ray-Tracing: Learning Surfaces and Reflectance for Relighting and View Synthesis
• Visual Tomography: Physically Faithful Volumetric Models of Partially Translucent Objects