Im2Struct: Recovering 3D Shape Structure from a Single RGB Image

Abstract

We propose to recover 3D shape structures from single RGB images, where structure refers to shape parts represented by cuboids and part relations encompassing connectivity and symmetry. Given a single 2D image with an object depicted, our goal is automatically recover a cuboid structure of the object parts as well as their mutual relations. We develop a convolutional-recursive auto-encoder comprised of structure parsing of a 2D image followed by structure recovering of a cuboid hierarchy. The encoder is achieved by a multi-scale convolutional network trained with the task of shape contour estimation, thereby learning to discern object structures in various forms and scales. The decoder fuses the features of the structure parsing network and the original image, and recursively decodes a hierarchy of cuboids. Since the decoder network is learned to recover part relations including connectivity and symmetry explicitly, the plausibility and generality of part structure recovery can be ensured. The two networks are jointly trained using the training data of contour-mask and cuboid structure pairs. Such pairs are generated by rendering stock 3D CAD models coming with part segmentation. Our method achieves unprecedentedly faithful and detailed recovery of diverse 3D part structures from single-view 2D images. We demonstrate two applications of our method including structure-guided completion of 3D volumes reconstructed from single-view images and structure-aware interactive editing of 2D images.

arXiv commentCVPR 2018

References in corpus (5)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• MatConvNet - Convolutional Neural Networks for MATLAB
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• A Point Set Generation Network for 3D Object Reconstruction from a Single Image
• 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks

Cited by in corpus (5)

• DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
• Neural Star Domain as Primitive Representation
• PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes
• Segmentation and Recovery of Superquadric Models using Convolutional Neural Networks
• Single Image 3D Object Estimation with Primitive Graph Networks