A Neural Rendering Framework for Free-Viewpoint Relighting
arXiv:1911.11530
Abstract
We present a novel Relightable Neural Renderer (RNR) for simultaneous view synthesis and relighting using multi-view image inputs. Existing neural rendering (NR) does not explicitly model the physical rendering process and hence has limited capabilities on relighting. RNR instead models image formation in terms of environment lighting, object intrinsic attributes, and light transport function (LTF), each corresponding to a learnable component. In particular, the incorporation of a physically based rendering process not only enables relighting but also improves the quality of view synthesis. Comprehensive experiments on synthetic and real data show that RNR provides a practical and effective solution for conducting free-viewpoint relighting.
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Semi-Supervised Classification with Graph Convolutional Networks
- Deep Convolutional Inverse Graphics Network
- Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
- Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
- Differentiable Surface Splatting for Point-based Geometry Processing
- MONet: Unsupervised Scene Decomposition and Representation
- Stereo Magnification: Learning View Synthesis using Multiplane Images
- Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
- DeepGCNs: Can GCNs Go as Deep as CNNs?
- Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
- RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
- Explorations in Homeomorphic Variational Auto-Encoding
- Deferred Neural Rendering: Image Synthesis using Neural Textures
- Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
- Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
- IGNOR: Image-guided Neural Object Rendering
- Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image
- Geometry-aware Deep Network for Single-Image Novel View Synthesis
- Neural Rerendering in the Wild
- Transformable Bottleneck Networks