Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

Abstract

We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.

arXiv commentCVPR 2018. The first two authors contributed equally to this work. Project page: http://pix3d.csail.mit.edu

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Deep Residual Learning for Image Recognition
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• A Large Dataset of Object Scans
• Unsupervised Learning of 3D Structure from Images
• A Point Set Generation Network for 3D Object Reconstruction from a Single Image
• Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
• Semantic Scene Completion from a Single Depth Image
• OctNet: Learning Deep 3D Representations at High Resolutions
• Viewpoints and Keypoints
• T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects

Cited by in corpus (21)

• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• 3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image
• 3D Shape Reconstruction from Vision and Touch
• Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
• Unsupervised Learning of Shape and Pose with Differentiable Point Clouds
• FroDO: From Detections to 3D Objects
• Learning Shape Priors for Single-View 3D Completion and Reconstruction
• Synthetic 3D Data Generation Pipeline for Geometric Deep Learning in Architecture
• 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers
• Zero in on Shape: A Generic 2D-3D Instance Similarity Metric learned from Synthetic Data
• A Simple and Scalable Shape Representation for 3D Reconstruction
• 3D-GMNet: Single-View 3D Shape Recovery as A Gaussian Mixture
• Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
• Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
• Active 3D Shape Reconstruction from Vision and Touch
• Review on 6D Object Pose Estimation with the focus on Indoor Scene Understanding
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Associative3D: Volumetric Reconstruction from Sparse Views
• Single Image 3D Object Estimation with Primitive Graph Networks
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction