ControlMat: A Controlled Generative Approach to Material Capture
arXiv:2309.01700 · doi:10.1145/3688830
Abstract
Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. Supplemental materials and additional details are available at: https://gvecchio.com/controlmat/.
References in corpus (13)
- Denoising Diffusion Probabilistic Models
- Diffusion Models Beat GANs on Image Synthesis
- Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
- Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks
- Generating Images with Perceptual Similarity Metrics based on Deep Networks
- MaterialGAN: Reflectance Capture using a Generative SVBRDF Model
- MatFormer: A Generative Model for Procedural Materials
- Guided Fine-Tuning for Large-Scale Material Transfer
- MatFusion: A Generative Diffusion Model for SVBRDF Capture
- PhotoMat: A Material Generator Learned from Single Flash Photos
- Generating Procedural Materials from Text or Image Prompts
- Node Graph Optimization Using Differentiable Proxies
- Network-to-Network Translation with Conditional Invertible Neural Networks