A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Abstract

Generation of 3D data by deep neural network has been attracting increasing attention in the research community. The majority of extant works resort to regular representations such as volumetric grids or collection of images; however, these representations obscure the natural invariance of 3D shapes under geometric transformations and also suffer from a number of other issues. In this paper we address the problem of 3D reconstruction from a single image, generating a straight-forward form of output -- point cloud coordinates. Along with this problem arises a unique and interesting issue, that the groundtruth shape for an input image may be ambiguous. Driven by this unorthodox output form and the inherent ambiguity in groundtruth, we design architecture, loss function and learning paradigm that are novel and effective. Our final solution is a conditional shape sampler, capable of predicting multiple plausible 3D point clouds from an input image. In experiments not only can our system outperform state-of-the-art methods on single image based 3d reconstruction benchmarks; but it also shows a strong performance for 3d shape completion and promising ability in making multiple plausible predictions.

The first two authors contributed equally

References in corpus (5)

• Conditional Generative Adversarial Nets
• Tutorial on Variational Autoencoders
• Stacked Hourglass Networks for Human Pose Estimation
• Unsupervised Learning of 3D Structure from Images
• 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Cited by in corpus (30)

• Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network
• Deep Level Sets: Implicit Surface Representations for 3D Shape Inference
• Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
• 3D Point Cloud Classification and Segmentation using 3D Modified Fisher Vector Representation for Convolutional Neural Networks
• The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics
• GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud
• DeepSpline: Data-Driven Reconstruction of Parametric Curves and Surfaces
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs
• PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image
• Learning to Synthesize a 4D RGBD Light Field from a Single Image
• NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping
• Learning Part Generation and Assembly for Structure-aware Shape Synthesis
• Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction
• Im2Struct: Recovering 3D Shape Structure from a Single RGB Image
• Unsupervised 3D Reconstruction from a Single Image via Adversarial Learning
• ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image
• DualSDF: Semantic Shape Manipulation using a Two-Level Representation
• Local Deep Implicit Functions for 3D Shape
• Mesh-based Autoencoders for Localized Deformation Component Analysis
• Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds
• Semantic Photometric Bundle Adjustment on Natural Sequences
• From Images to 3D Shape Attributes
• Object-Centric Photometric Bundle Adjustment with Deep Shape Prior
• AdvectiveNet: An Eulerian-Lagrangian Fluidic reservoir for Point Cloud Processing
• Learning View Priors for Single-view 3D Reconstruction
• Latent Space Representation for Shape Analysis and Learning
• TreeGCN-ED: Encoding Point Cloud using a Tree-Structured Graph Network
• Absolute 3D Pose Estimation and Length Measurement of Severely Deformed Fish from Monocular Videos in Longline Fishing