Self-supervised Learning of 3D Objects from Natural Images

Abstract

We present a method to learn single-view reconstruction of the 3D shape, pose, and texture of objects from categorized natural images in a self-supervised manner. Since this is a severely ill-posed problem, carefully designing a training method and introducing constraints are essential. To avoid the difficulty of training all elements at the same time, we propose training category-specific base shapes with fixed pose distribution and simple textures first, and subsequently training poses and textures using the obtained shapes. Another difficulty is that shapes and backgrounds sometimes become excessively complicated to mistakenly reconstruct textures on object surfaces. To suppress it, we propose using strong regularization and constraints on object surfaces and background images. With these two techniques, we demonstrate that we can use natural image collections such as CIFAR-10 and PASCAL objects for training, which indicates the possibility to realize 3D object reconstruction on diverse object categories beyond synthetic datasets.

Technical report. Project page: http://hiroharu-kato.com/projects_en/cifar10_3d.html

References in corpus (5)

• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
• Learning to Infer Implicit Surfaces without 3D Supervision
• Learning to Reconstruct Shapes from Unseen Classes
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

Cited by in corpus (2)

• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes