Learning to Reconstruct Shapes from Unseen Classes

Abstract

From a single image, humans are able to perceive the full 3D shape of an object by exploiting learned shape priors from everyday life. Contemporary single-image 3D reconstruction algorithms aim to solve this task in a similar fashion, but often end up with priors that are highly biased by training classes. Here we present an algorithm, Generalizable Reconstruction (GenRe), designed to capture more generic, class-agnostic shape priors. We achieve this with an inference network and training procedure that combine 2.5D representations of visible surfaces (depth and silhouette), spherical shape representations of both visible and non-visible surfaces, and 3D voxel-based representations, in a principled manner that exploits the causal structure of how 3D shapes give rise to 2D images. Experiments demonstrate that GenRe performs well on single-view shape reconstruction, and generalizes to diverse novel objects from categories not seen during training.

NeurIPS 2018 (Oral). The first two authors contributed equally to this paper. Project page: http://genre.csail.mit.edu/

Cited by in corpus (22)

• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion
• pixelNeRF: Neural Radiance Fields from One or Few Images
• Single-view 3D Mesh Reconstruction for Seen and Unseen Categories
• AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• Higher-Order Function Networks for Learning Composable 3D Object Representations
• Inferring Point Clouds from Single Monocular Images by Depth Intermediation
• Amodal 3D Reconstruction for Robotic Manipulation via Stability and Connectivity
• Self-supervised Learning of 3D Objects from Natural Images
• Predicting Camera Viewpoint Improves Cross-dataset Generalization for 3D Human Pose Estimation
• Learning Manifold Patch-Based Representations of Man-Made Shapes
• Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Towards Panoptic 3D Parsing for Single Image in the Wild
• SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images
• Single-View 3D Object Reconstruction from Shape Priors in Memory
• Toward 3D Object Reconstruction from Stereo Images
• Interactive Annotation of 3D Object Geometry using 2D Scribbles
• Surface HOF: Surface Reconstruction from a Single Image Using Higher Order Function Networks
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction
• Associative3D: Volumetric Reconstruction from Sparse Views