MarrNet: 3D Shape Reconstruction via 2.5D Sketches

Abstract

3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.

arXiv commentNIPS 2017. The first two authors contributed equally to this paper. Project page: http://marrnet.csail.mit.edu

Cited by in corpus (41)

• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
• Dense 3D Object Reconstruction from a Single Depth View
• Self-Supervised Intrinsic Image Decomposition
• Learning to Infer and Execute 3D Shape Programs
• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion
• RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
• GIFS: Neural Implicit Function for General Shape Representation
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• Im2Avatar: Colorful 3D Reconstruction from a Single Image
• Convolutional Generation of Textured 3D Meshes
• 3D Shape Reconstruction from Vision and Touch
• Single Day Outdoor Photometric Stereo
• Learning Pose-invariant 3D Object Reconstruction from Single-view Images
• Grasping Field: Learning Implicit Representations for Human Grasps
• SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
• Points2Pix: 3D Point-Cloud to Image Translation using conditional Generative Adversarial Networks
• 3D Shape Perception from Monocular Vision, Touch, and Shape Priors
• Transformable Bottleneck Networks
• StrobeNet: Category-Level Multiview Reconstruction of Articulated Objects
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• Learning Manifold Patch-Based Representations of Man-Made Shapes
• Object-Centric Multi-View Aggregation
• 3D-GMNet: Single-View 3D Shape Recovery as A Gaussian Mixture
• Semantic Photometric Bundle Adjustment on Natural Sequences
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Toward 3D Object Reconstruction from Stereo Images
• Deep Implicit Templates for 3D Shape Representation
• Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
• Single-view 3D Body and Cloth Reconstruction under Complex Poses
• Diverse Plausible Shape Completions from Ambiguous Depth Images
• Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
• Single-View 3D Object Reconstruction from Shape Priors in Memory
• Look, Cast and Mold: Learning 3D Shape Manifold from Single-view Synthetic Data
• D-OccNet: Detailed 3D Reconstruction Using Cross-Domain Learning
• Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate
• FusionNet: Incorporating Shape and Texture for Abnormality Detection in 3D Abdominal CT Scans
• DiverseNet: When One Right Answer is not Enough
• StylePart: Image-based Shape Part Manipulation
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction
• Learning to Reconstruct and Segment 3D Objects