DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

Abstract

We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning-based approaches for inverse graphics adopt rasterization-based renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIBR++, a hybrid differentiable renderer which supports these photorealistic effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths -- speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIBR++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry, reflectance and lighting prediction from a single image without requiring any ground-truth. We experimentally demonstrate that our approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches and showcase several artistic applications including material editing and relighting.

References in corpus (5)

• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• Neural Reflectance Fields for Appearance Acquisition
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research