Unsupervised 3D Shape Learning from Image Collections in the Wild

Abstract

We present a method to learn the 3D surface of objects directly from a collection of images. Previous work achieved this capability by exploiting additional manual annotation, such as object pose, 3D surface templates, temporal continuity of videos, manually selected landmarks, and foreground/background masks. In contrast, our method does not make use of any such annotation. Rather, it builds a generative model, a convolutional neural network, which, given a noise vector sample, outputs the 3D surface and texture of an object and a background image. These 3 components combined with an additional random viewpoint vector are then fed to a differential renderer to produce a view of the sampled object and background. Our general principle is that if the output of the renderer, the generated image, is realistic, then its input, the generated 3D and texture, should also be realistic. To achieve realism, the generative model is trained adversarially against a discriminator that tries to distinguish between the output of the renderer and real images from the given data set. Moreover, our generative model can be paired with an encoder and trained as an autoencoder, to automatically extract the 3D shape, texture and pose of the object in an image. Our trained generative model and encoder show promising results both on real and synthetic data, which demonstrate for the first time that fully unsupervised 3D learning from image collections is possible.

References in corpus (9)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Improved Training of Wasserstein GANs
• Disentangling factors of variation in deep representations using adversarial training
• Unsupervised Learning of Depth and Ego-Motion from Video
• Deep Learning Face Attributes in the Wild
• Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
• 3D Shape Induction from 2D Views of Multiple Objects
• Unsupervised Training for 3D Morphable Model Regression

Cited by in corpus (4)

• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation
• Self-supervised Single-view 3D Reconstruction via Semantic Consistency
• Photo-Geometric Autoencoding to Learn 3D Objects from Unlabelled Images