3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks

Abstract

We propose a method for reconstructing 3D shapes from 2D sketches in the form of line drawings. Our method takes as input a single sketch, or multiple sketches, and outputs a dense point cloud representing a 3D reconstruction of the input sketch(es). The point cloud is then converted into a polygon mesh. At the heart of our method lies a deep, encoder-decoder network. The encoder converts the sketch into a compact representation encoding shape information. The decoder converts this representation into depth and normal maps capturing the underlying surface from several output viewpoints. The multi-view maps are then consolidated into a 3D point cloud by solving an optimization problem that fuses depth and normals across all viewpoints. Based on our experiments, compared to other methods, such as volumetric networks, our architecture offers several advantages, including more faithful reconstruction, higher output surface resolution, better preservation of topology and shape structure.

3DV 2017 (oral)

References in corpus (6)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Sketch-based 3D Shape Retrieval using Convolutional Neural Networks
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Learning Representations for Automatic Colorization

Cited by in corpus (10)

• Tangent Convolutions for Dense Prediction in 3D
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• Coherent Point Drift Networks: Unsupervised Learning of Non-Rigid Point Set Registration
• Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction
• Early-Phase Performance-Driven Design using Generative Models
• CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual Reality
• Im2Struct: Recovering 3D Shape Structure from a Single RGB Image
• Photo Wake-Up: 3D Character Animation from a Single Photo
• 3D Shape Reconstruction from a Single 2D Image via 2D-3D Self-Consistency
• End-to-end 3D shape inverse rendering of different classes of objects from a single input image