MaterialGAN: Reflectance Capture using a Generative SVBRDF Model

Abstract

We address the problem of reconstructing spatially-varying BRDFs from a small set of image measurements. This is a fundamentally under-constrained problem, and previous work has relied on using various regularization priors or on capturing many images to produce plausible results. In this work, we present MaterialGAN, a deep generative convolutional network based on StyleGAN2, trained to synthesize realistic SVBRDF parameter maps. We show that MaterialGAN can be used as a powerful material prior in an inverse rendering framework: we optimize in its latent representation to generate material maps that match the appearance of the captured images when rendered. We demonstrate this framework on the task of reconstructing SVBRDFs from images captured under flash illumination using a hand-held mobile phone. Our method succeeds in producing plausible material maps that accurately reproduce the target images, and outperforms previous state-of-the-art material capture methods in evaluations on both synthetic and real data. Furthermore, our GAN-based latent space allows for high-level semantic material editing operations such as generating material variations and material morphing.

arXiv comment13 pages, 16 figures. Siggraph Asia 2020

References in corpus (3)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks

Cited by in corpus (17)

• MatFormer: A Generative Model for Procedural Materials
• An Inverse Procedural Modeling Pipeline for SVBRDF Maps
• ControlMat: A Controlled Generative Approach to Material Capture
• MatFusion: A Generative Diffusion Model for SVBRDF Capture
• MatSynth: A Modern PBR Materials Dataset
• MatFuse: Controllable Material Generation with Diffusion Models
• PhotoMat: A Material Generator Learned from Single Flash Photos
• SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps
• Generating Procedural Materials from Text or Image Prompts
• Node Graph Optimization Using Differentiable Proxies
• UMat: Uncertainty-Aware Single Image High Resolution Material Capture
• Learning to Learn and Sample BRDFs
• Neural Photometry-guided Visual Attribute Transfer
• Deep scene-scale material estimation from multi-view indoor captures
• MaterialPicker: Multi-Modal DiT-Based Material Generation
• MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models
• Neural Differential Appearance Equations