Neural Light Transport for Relighting and View Synthesis

Abstract

The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT that is embedded in the space of a texture atlas of known geometric properties, and model all non-diffuse and global LT as residuals added to a physically-accurate diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination, while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse, previously seen observations. Qualitative and quantitative experiments demonstrate that our neural LT (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without separate treatment for both problems that prior work requires.

arXiv commentCamera-ready version for TOG 2021. Project Page: http://nlt.csail.mit.edu/

References in corpus (5)

• Neural Volumes: Learning Dynamic Renderable Volumes from Images
• Deep Appearance Models for Face Rendering
• Single Image Portrait Relighting
• Deep Shading: Convolutional Neural Networks for Screen-Space Shading
• LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering

Cited by in corpus (12)

• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
• DeepCloth: Neural Garment Representation for Shape and Style Editing
• Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation
• Automated Educational Question Generation at Different Bloom's Skill Levels using Large Language Models: Strategies and Evaluation
• Relighting Neural Radiance Fields with Shadow and Highlight Hints
• Hybrid Skip: A Biologically Inspired Skip Connection for the UNet Architecture
• DifFRelight: Diffusion-Based Facial Performance Relighting
• Interactive Rendering of Relightable and Animatable Gaussian Avatars
• Hierarchical Vectorization for Portrait Images
• Neural BSSRDF: Object Appearance Representation Including Heterogeneous Subsurface Scattering
• Manifold Sampling for Differentiable Uncertainty in Radiance Fields
• GSHeadRelight: Fast Relightability for 3D Gaussian Head Synthesis