Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
arXiv:2010.09125
Abstract
Differentiable rendering has paved the way to training neural networks to perform "inverse graphics" tasks such as predicting 3D geometry from monocular photographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice. Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies. We further showcase the disentangled GAN as a controllable 3D "neural renderer", complementing traditional graphics renderers.
arXiv commentAccepted to ICLR 2021 as an Oral paper
References in corpus (12)
- Generative Adversarial Text to Image Synthesis
- Invertible Conditional GANs for image editing
- Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
- Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
- Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
- Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
- Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
- Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer
- Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
- Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
- Self-supervised Learning of 3D Objects from Natural Images