NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination

Abstract

We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu/xiuming/projects/nerfactor/.

Camera-ready version for SIGGRAPH Asia 2021. Project Page: https://people.csail.mit.edu/xiuming/projects/nerfactor/

References in corpus (3)

• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Neural Reflectance Fields for Appearance Acquisition
• PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting

Cited by in corpus (26)

• Neural 3D Reconstruction in the Wild
• Relighting Neural Radiance Fields with Shadow and Highlight Hints
• 3D Gaussian Splatting with Deferred Reflection
• Fast Learning Radiance Fields by Shooting Much Fewer Rays
• EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations
• GS^3: Efficient Relighting with Triple Gaussian Splatting
• DifFRelight: Diffusion-Based Facial Performance Relighting
• ZS-SRT: An Efficient Zero-Shot Super-Resolution Training Method for Neural Radiance Fields
• TetWeave: Isosurface Extraction using On-The-Fly Delaunay Tetrahedral Grids for Gradient-Based Mesh Optimization
• Interactive Rendering of Relightable and Animatable Gaussian Avatars
• VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization
• Inverse Global Illumination using a Neural Radiometric Prior
• Structured Generative Models for Scene Understanding
• GS-ROR${}^2$: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstruction
• Neural BSSRDF: Object Appearance Representation Including Heterogeneous Subsurface Scattering
• DPCS: Path Tracing-Based Differentiable Projector-Camera Systems
• Manifold Sampling for Differentiable Uncertainty in Radiance Fields
• MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Field
• P2I-NET: Mapping Camera Pose to Image via Adversarial Learning for New View Synthesis in Real Indoor Environments
• Real-Time Rendering of Arbitrary Surface Geometries using Learnt Transfer
• GS-ProCams: Gaussian Splatting-based Projector-Camera Systems
• NoPe-NeRF++: Local-to-Global Optimization of NeRF with No Pose Prior
• Advances in Neural 3D Mesh Texturing: A Survey
• TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenes
• Intrinsic decomposition and editing of 3D Gaussian splats