3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image

Abstract

3D reconstruction from single view images is an ill-posed problem. Inferring the hidden regions from self-occluded images is both challenging and ambiguous. We propose a two-pronged approach to address these issues. To better incorporate the data prior and generate meaningful reconstructions, we propose 3D-LMNet, a latent embedding matching approach for 3D reconstruction. We first train a 3D point cloud auto-encoder and then learn a mapping from the 2D image to the corresponding learnt embedding. To tackle the issue of uncertainty in the reconstruction, we predict multiple reconstructions that are consistent with the input view. This is achieved by learning a probablistic latent space with a novel view-specific diversity loss. Thorough quantitative and qualitative analysis is performed to highlight the significance of the proposed approach. We outperform state-of-the-art approaches on the task of single-view 3D reconstruction on both real and synthetic datasets while generating multiple plausible reconstructions, demonstrating the generalizability and utility of our approach.

arXiv commentAccepted at BMVC 2018; Codes are available at https://github.com/val-iisc/3d-lmnet

References in corpus (7)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Tutorial on Variational Autoencoders
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• PU-Net: Point Cloud Upsampling Network
• PPFNet: Global Context Aware Local Features for Robust 3D Point Matching
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

Cited by in corpus (14)

• Inferring Point Clouds from Single Monocular Images by Depth Intermediation
• ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds
• Learning Manifold Patch-Based Representations of Man-Made Shapes
• CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision
• One-stage Shape Instantiation from a Single 2D Image to 3D Point Cloud
• Structure-Attentioned Memory Network for Monocular Depth Estimation
• Local Aggressive Adversarial Attacks on 3D Point Cloud
• SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images
• Look, Cast and Mold: Learning 3D Shape Manifold from Single-view Synthetic Data
• Discrete Point Flow Networks for Efficient Point Cloud Generation
• D-OccNet: Detailed 3D Reconstruction Using Cross-Domain Learning
• Silhouette Guided Point Cloud Reconstruction beyond Occlusion
• Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images
• Neural Implicit 3D Shapes from Single Images with Spatial Patterns