Self-supervised Single-view 3D Reconstruction via Semantic Consistency

Abstract

We learn a self-supervised, single-view 3D reconstruction model that predicts the 3D mesh shape, texture and camera pose of a target object with a collection of 2D images and silhouettes. The proposed method does not necessitate 3D supervision, manually annotated keypoints, multi-view images of an object or a prior 3D template. The key insight of our work is that objects can be represented as a collection of deformable parts, and each part is semantically coherent across different instances of the same category (e.g., wings on birds and wheels on cars). Therefore, by leveraging self-supervisedly learned part segmentation of a large collection of category-specific images, we can effectively enforce semantic consistency between the reconstructed meshes and the original images. This significantly reduces ambiguities during joint prediction of shape and camera pose of an object, along with texture. To the best of our knowledge, we are the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints. Thus our model can easily generalize to various object categories without such labels, e.g., horses, penguins, etc. Through a variety of experiments on several categories of deformable and rigid objects, we demonstrate that our unsupervised method performs comparably if not better than existing category-specific reconstruction methods learned with supervision.

arXiv commentCodes and other resources will be released at: https://sites.google.com/view/unsup-mesh/home

References in corpus (9)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Learning to Infer Implicit Surfaces without 3D Supervision
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image
• Unsupervised 3D Shape Learning from Image Collections in the Wild
• Photo-Geometric Autoencoding to Learn 3D Objects from Unlabelled Images

Cited by in corpus (2)

• Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches
• Unsupervised Part Segmentation through Disentangling Appearance and Shape