Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data

Abstract

Recent work has shown the ability to learn generative models for 3D shapes from only unstructured 2D images. However, training such models requires differentiating through the rasterization step of the rendering process, therefore past work has focused on developing bespoke rendering models which smooth over this non-differentiable process in various ways. Such models are thus unable to take advantage of the photo-realistic, fully featured, industrial renderers built by the gaming and graphics industry. In this paper we introduce the first scalable training technique for 3D generative models from 2D data which utilizes an off-the-shelf non-differentiable renderer. To account for the non-differentiability, we introduce a proxy neural renderer to match the output of the non-differentiable renderer. We further propose discriminator output matching to ensure that the neural renderer learns to smooth over the rasterization appropriately. We evaluate our model on images rendered from our generated 3D shapes, and show that our model can consistently learn to generate better shapes than existing models when trained with exclusively unstructured 2D images.

8 pages paper, 3 pages references, 18 pages appendix

References in corpus (13)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Learning Representations and Generative Models for 3D Point Clouds
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Learning to Infer Implicit Surfaces without 3D Supervision
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Cited by in corpus (6)

• Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
• Inferring Implicit 3D Representations from Human Figures on Pictorial Maps
• Lifting 2D StyleGAN for 3D-Aware Face Generation
• 3D-Aware Semantic-Guided Generative Model for Human Synthesis