InverseRenderNet: Learning single image inverse rendering

Abstract

We show how to train a fully convolutional neural network to perform inverse rendering from a single, uncontrolled image. The network takes an RGB image as input, regresses albedo and normal maps from which we compute lighting coefficients. Our network is trained using large uncontrolled image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision: 1. We learn a statistical natural illumination prior, 2. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering.

References in corpus (17)

• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Deep Convolutional Inverse Graphics Network
• Stacked Hourglass Networks for Human Pose Estimation
• SfM-Net: Learning of Structure and Motion from Video
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• End-to-End Learning of Geometry and Context for Deep Stereo Regression
• Unsupervised Learning of Depth and Ego-Motion from Video
• Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks
• Deep Ordinal Regression Network for Monocular Depth Estimation
• Single-Image Depth Perception in the Wild
• Self-Supervised Intrinsic Image Decomposition
• MegaDepth: Learning Single-View Depth Prediction from Internet Photos
• Neural Face Editing with Intrinsic Image Disentangling
• Learning Intrinsic Image Decomposition from Watching the World
• Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression
• Deep Hybrid Real and Synthetic Training for Intrinsic Decomposition