Dense 3D Object Reconstruction from a Single Depth View

Abstract

In this paper, we propose a novel approach, 3D-RecGAN++, which reconstructs the complete 3D structure of a given object from a single arbitrary depth view using generative adversarial networks. Unlike existing work which typically requires multiple views of the same object or class labels to recover the full 3D geometry, the proposed 3D-RecGAN++ only takes the voxel grid representation of a depth view of the object as input, and is able to generate the complete 3D occupancy grid with a high resolution of 256^3 by recovering the occluded/missing regions. The key idea is to combine the generative capabilities of autoencoders and the conditional Generative Adversarial Networks (GAN) framework, to infer accurate and fine-grained 3D structures of objects in high-dimensional voxel space. Extensive experiments on large synthetic datasets and real-world Kinect datasets show that the proposed 3D-RecGAN++ significantly outperforms the state of the art in single view 3D object reconstruction, and is able to reconstruct unseen types of objects.

arXiv commentTPAMI 2018. Code and data are available at: https://github.com/Yang7879/3D-RecGAN-extended. This article extends from arXiv:1708.07969

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• ShapeNet: An Information-Rich 3D Model Repository
• Generative Adversarial Text to Image Synthesis
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Deep Convolutional Inverse Graphics Network
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Learning a Multi-View Stereo Machine
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Reconstructing Vechicles from a Single Image: Shape Priors for Road Scene Understanding

Cited by in corpus (25)

• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
• Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images
• Learning 3D Shape Completion under Weak Supervision
• PCN: Point Completion Network
• Learning Category-Specific Mesh Reconstruction from Image Collections
• A Large RGB-D Dataset for Semi-supervised Monocular Depth Estimation
• SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation
• SoftPool++: An Encoder-Decoder Network for Point Cloud Completion
• DeepHuman: 3D Human Reconstruction from a Single Image
• Efficient Semantic Scene Completion Network with Spatial Group Convolution
• GANs-NQM: A Generative Adversarial Networks based No Reference Quality Assessment Metric for RGB-D Synthesized Views
• Multimodal Shape Completion via Conditional Generative Adversarial Networks
• 3D Shape Reconstruction from Free-Hand Sketches
• SoftPoolNet: Shape Descriptor for Point Cloud Completion and Classification
• Deep Octree-based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completion
• Where Does It End? -- Reasoning About Hidden Surfaces by Object Intersection Constraints
• 3D Semantic Scene Completion from a Single Depth Image using Adversarial Training
• Self-supervised Latent Space Optimization with Nebula Variational Coding
• Toward 3D Object Reconstruction from Stereo Images
• Diverse Plausible Shape Completions from Ambiguous Depth Images
• Synthesizing 3D Shapes from Silhouette Image Collections using Multi-projection Generative Adversarial Networks
• 3D Facial Geometry Recovery from a Depth View with Attention Guided Generative Adversarial Network
• Learning to Reconstruct and Segment 3D Objects
• Silhouette Guided Point Cloud Reconstruction beyond Occlusion