Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision

Abstract

We present a unified framework tackling two problems: class-specific 3D reconstruction from a single image, and generation of new 3D shape samples. These tasks have received considerable attention recently; however, existing approaches rely on 3D supervision, annotation of 2D images with keypoints or poses, and/or training with multiple views of each object instance. Our framework is very general: it can be trained in similar settings to these existing approaches, while also supporting weaker supervision scenarios. Importantly, it can be trained purely from 2D images, without ground-truth pose annotations, and with a single view per instance. We employ meshes as an output representation, instead of voxels used in most prior work. This allows us to exploit shading information during training, which previous 2D-supervised methods cannot. Thus, our method can learn to generate and reconstruct concave object classes. We evaluate our approach on synthetic data in various settings, showing that (i) it learns to disentangle shape from pose; (ii) using shading in the loss improves performance; (iii) our model is comparable or superior to state-of-the-art voxel-based approaches on quantitative metrics, while producing results that are visually more pleasing; (iv) it still performs well when given supervision weaker than in prior works.

arXiv commentBMVC 2018 (Oral). Differentiable renderer available at https://github.com/pmh47/dirt

References in corpus (3)

• Auto-Encoding Variational Bayes
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image

Cited by in corpus (14)

• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• 3D Shape Reconstruction from Vision and Touch
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
• Cerberus: A Multi-headed Derenderer
• Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images using a View-based Representation
• Unsupervised 3D Shape Learning from Image Collections in the Wild
• SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering
• From Image Collections to Point Clouds with Self-supervised Shape and Pose Networks
• Discrete Point Flow Networks for Efficient Point Cloud Generation
• End-to-End Optimization of Scene Layout
• Learning Equivariant Representations
• AlgoNet: $C^{\infty }$ Smooth Algorithmic Neural Networks