Learning Pose-invariant 3D Object Reconstruction from Single-view Images

Abstract

Learning to reconstruct 3D shapes using 2D images is an active research topic, with benefits of not requiring expensive 3D data. However, most work in this direction requires multi-view images for each object instance as training supervision, which oftentimes does not apply in practice. In this paper, we relax the common multi-view assumption and explore a more challenging yet more realistic setup of learning 3D shape from only single-view images. The major difficulty lies in insufficient constraints that can be provided by single view images, which leads to the problem of pose entanglement in learned shape space. As a result, reconstructed shapes vary along input pose and have poor accuracy. We address this problem by taking a novel domain adaptation perspective, and propose an effective adversarial domain confusion method to learn pose-disentangled compact shape space. Experiments on single-view reconstruction show effectiveness in solving pose entanglement, and the proposed method achieves on-par reconstruction accuracy with state-of-the-art with higher efficiency.

arXiv commentunder review, code available at https://github.com/bomb2peng/learn3D

References in corpus (14)

• Auto-Encoding Variational Bayes
• ShapeNet: An Information-Rich 3D Model Repository
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Adversarial Discriminative Domain Adaptation
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Learning a Multi-View Stereo Machine
• Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• Domain-Adversarial Training of Neural Networks
• Learning 3D Shape Completion under Weak Supervision
• Nonlinear 3D Face Morphable Model