3D Object Reconstruction from a Single Depth View with Adversarial Learning

Abstract

In this paper, we propose a novel 3D-RecGAN approach, which reconstructs the complete 3D structure of a given object from a single arbitrary depth view using generative adversarial networks. Unlike the existing work which typically requires multiple views of the same object or class labels to recover the full 3D geometry, the proposed 3D-RecGAN only takes the voxel grid representation of a depth view of the object as input, and is able to generate the complete 3D occupancy grid by filling in the occluded/missing regions. The key idea is to combine the generative capabilities of autoencoders and the conditional Generative Adversarial Networks (GAN) framework, to infer accurate and fine-grained 3D structures of objects in high-dimensional voxel space. Extensive experiments on large synthetic datasets show that the proposed 3D-RecGAN significantly outperforms the state of the art in single view 3D object reconstruction, and is able to reconstruct unseen types of objects. Our code and data are available at: https://github.com/Yang7879/3D-RecGAN.

ICCV Workshops 2017

References in corpus (12)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Improved Training of Wasserstein GANs
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Fully Convolutional Networks for Semantic Segmentation
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• 3D ShapeNets: A Deep Representation for Volumetric Shapes
• A Point Set Generation Network for 3D Object Reconstruction from a Single Image
• Semantic Scene Completion from a Single Depth Image
• 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks
• OctNetFusion: Learning Depth Fusion from Data
• 3D Shape Induction from 2D Views of Multiple Objects

Cited by in corpus (10)

• PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention
• A survey of Object Classification and Detection based on 2D/3D data
• Deep-SLAM++: Object-level RGBD SLAM based on class-specific deep shape priors
• Inferring Point Clouds from Single Monocular Images by Depth Intermediation
• 3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations
• KTNet: Knowledge Transfer for Unpaired 3D Shape Completion
• ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image
• A Variational Feature Encoding Method of 3D Object for Probabilistic Semantic SLAM
• A Variational Observation Model of 3D Object for Probabilistic Semantic SLAM
• Point Cloud Colorization Based on Densely Annotated 3D Shape Dataset