Unsupervised Generative 3D Shape Learning from Natural Images

Abstract

In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way. For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image generation is split into 2 stages. In the first stage a generator network outputs 3D objects. In the second, a differentiable renderer produces an image of the 3D objects from random viewpoints. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to overcome them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.

The paper is under review

References in corpus (4)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Learning Representations and Generative Models for 3D Point Clouds
• Unsupervised Training for 3D Morphable Model Regression
• Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision

Cited by in corpus (7)

• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
• Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions
• Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
• Unsupervised object-centric video generation and decomposition in 3D
• Lifting 2D StyleGAN for 3D-Aware Face Generation
• Unsupervised Learning of Depth and Depth-of-Field Effect from Natural Images with Aperture Rendering Generative Adversarial Networks