Sketch-based 3D Shape Retrieval using Convolutional Neural Networks

Abstract

Retrieving 3D models from 2D human sketches has received considerable attention in the areas of graphics, image retrieval, and computer vision. Almost always in state of the art approaches a large amount of "best views" are computed for 3D models, with the hope that the query sketch matches one of these 2D projections of 3D models using predefined features. We argue that this two stage approach (view selection -- matching) is pragmatic but also problematic because the "best views" are subjective and ambiguous, which makes the matching inputs obscure. This imprecise nature of matching further makes it challenging to choose features manually. Instead of relying on the elusive concept of "best views" and the hand-crafted features, we propose to define our views using a minimalism approach and learn features for both sketches and views. Specifically, we drastically reduce the number of views to only two predefined directions for the whole dataset. Then, we learn two Siamese Convolutional Neural Networks (CNNs), one for the views and one for the sketches. The loss function is defined on the within-domain as well as the cross-domain similarities. Our experiments on three benchmark datasets demonstrate that our method is significantly better than state of the art approaches, and outperforms them in all conventional metrics.

CVPR 2015

Cited by in corpus (27)

• Recent Advance in Content-based Image Retrieval: A Literature Survey
• 3D Sketching using Multi-View Deep Volumetric Prediction
• Generalisation and Sharing in Triplet Convnets for Sketch based Visual Search
• Triplet-Center Loss for Multi-View 3D Object Retrieval
• Sketch-a-Net that Beats Humans
• Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
• SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis
• 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks
• Deep Learning for Free-Hand Sketch: A Survey
• Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval
• Localizing and Orienting Street Views Using Overhead Imagery
• Sketch-R2CNN: An Attentive Network for Vector Sketch Recognition
• Zero-Shot Sketch-Image Hashing
• Cartoonish sketch-based face editing in videos using identity deformation transfer
• Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition
• Learning in High-Dimensional Multimedia Data: The State of the Art
• SketchyScene: Richly-Annotated Scene Sketches
• Cross-modal Subspace Learning for Fine-grained Sketch-based Image Retrieval
• LUCSS: Language-based User-customized Colourization of Scene Sketches
• Multiple Discrimination and Pairwise CNN for View-based 3D Object Retrieval
• Towards an All-Purpose Content-Based Multimedia Information Retrieval System
• Instance-level Sketch-based Retrieval by Deep Triplet Classification Siamese Network
• Multi-Modal Retrieval using Graph Neural Networks
• Image Generation from Sketch Constraint Using Contextual GAN
• Deep Shape Matching
• Deep Cross-modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-based 3D Shape Retrieval
• Semi-supervised Learning for Convolutional Neural Networks via Online Graph Construction