Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

Abstract

In this paper, we develop novel, efficient 2D encodings for 3D geometry, which enable reconstructing full 3D shapes from a single image at high resolution. The key idea is to pose 3D shape reconstruction as a 2D prediction problem. To that end, we first develop a simple baseline network that predicts entire voxel tubes at each pixel of a reference view. By leveraging well-proven architectures for 2D pixel-prediction tasks, we attain state-of-the-art results, clearly outperforming purely voxel-based approaches. We scale this baseline to higher resolutions by proposing a memory-efficient shape encoding, which recursively decomposes a 3D shape into nested shape layers, similar to the pieces of a Matryoshka doll. This allows reconstructing highly detailed shapes with complex topology, as demonstrated in extensive experiments; we clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components. Our Matryoshka networks further enable reconstructing shapes from IDs or shape similarity, as well as shape sampling.

arXiv commentPublished at the Conference on Computer Vision and Pattern Recognition (CVPR 2018)

Cited by in corpus (26)

• Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images
• Neural Reflectance Fields for Appearance Acquisition
• Deep Level Sets: Implicit Surface Representations for 3D Shape Inference
• MeshSDF: Differentiable Iso-Surface Extraction
• Optimal Pose and Shape Estimation for Category-level 3D Object Perception
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers
• Cerberus: A Multi-headed Derenderer
• Predicting Camera Viewpoint Improves Cross-dataset Generalization for 3D Human Pose Estimation
• PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes
• GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision
• Implicitly Defined Layers in Neural Networks
• A Simple and Scalable Shape Representation for 3D Reconstruction
• 3D Shape Reconstruction from a Single 2D Image via 2D-3D Self-Consistency
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Learning View Priors for Single-view 3D Reconstruction
• Toward 3D Object Reconstruction from Stereo Images
• Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
• DmifNet:3D Shape Reconstruction Based on Dynamic Multi-Branch Information Fusion
• Learning 3D Part Assembly from a Single Image
• LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction
• PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction
• Domain Adaptation for Real-World Single View 3D Reconstruction
• Meta Deformation Network: Meta Functionals for Shape Correspondence
• Associative3D: Volumetric Reconstruction from Sparse Views
• What Do Single-view 3D Reconstruction Networks Learn?