3D Sketching using Multi-View Deep Volumetric Prediction

Abstract

Sketch-based modeling strives to bring the ease and immediacy of drawing to the 3D world. However, while drawings are easy for humans to create, they are very challenging for computers to interpret due to their sparsity and ambiguity. We propose a data-driven approach that tackles this challenge by learning to reconstruct 3D shapes from one or more drawings. At the core of our approach is a deep convolutional neural network (CNN) that predicts occupancy of a voxel grid from a line drawing. This CNN provides us with an initial 3D reconstruction as soon as the user completes a single drawing of the desired shape. We complement this single-view network with an updater CNN that refines an existing prediction given a new drawing of the shape created from a novel viewpoint. A key advantage of our approach is that we can apply the updater iteratively to fuse information from an arbitrary number of viewpoints, without requiring explicit stroke correspondences between the drawings. We train both CNNs by rendering synthetic contour drawings from hand-modeled shape collections as well as from procedurally-generated abstract shapes. Finally, we integrate our CNNs in a minimal modeling interface that allows users to seamlessly draw an object, rotate it to see its 3D reconstruction, and refine it by re-drawing from another vantage point using the 3D reconstruction as guidance. The main strengths of our approach are its robustness to freehand bitmap drawings, its ability to adapt to different object categories, and the continuum it offers between single-view and multi-view sketch-based modeling.

arXiv commentSee our accompanying video on https://youtu.be/DGIYzmlm2pQ, networks and databases available at https://ns.inria.fr/d3/3DSketching/. To appear in PACMCGIT

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Fully Convolutional Networks for Semantic Segmentation
• ShapeNet: An Information-Rich 3D Model Repository
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• Sketch-based 3D Shape Retrieval using Convolutional Neural Networks
• DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling
• Photographic Image Synthesis with Cascaded Refinement Networks
• Sketch-to-Design: Context-based Part Assembly
• Interactive 3D Modeling with a Generative Adversarial Network

Cited by in corpus (20)

• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• DeepSketchHair: Deep Sketch-based 3D Hair Modeling
• Deep Sketch-Based Modeling: Tips and Tricks
• TreeSketchNet: From Sketch To 3D Tree Parameters Generation
• Coherent Point Drift Networks: Unsupervised Learning of Non-Rigid Point Set Registration
• DeepPatent2: A Large-Scale Benchmarking Corpus for Technical Drawing Understanding
• Neural UpFlow: A Scene Flow Learning Approach to Increase the Apparent Resolution of Particle-Based Liquids
• 3D Shape Reconstruction from Free-Hand Sketches
• Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art
• Learning Manifold Patch-Based Representations of Man-Made Shapes
• GeoCode: Interpretable Shape Programs
• Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches
• Editing Conditional Radiance Fields
• Sketch2CAD: Sequential CAD Modeling by Sketching in Context
• 3D Shape Reconstruction from a Single 2D Image via 2D-3D Self-Consistency
• Inferring Implicit 3D Representations from Human Figures on Pictorial Maps
• SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence
• SHAD3S: A model to Sketch, Shade and Shadow
• SPARE3D: A Dataset for SPAtial REasoning on Three-View Line Drawings
• Interactive Annotation of 3D Object Geometry using 2D Scribbles