Generalisation and Sharing in Triplet Convnets for Sketch based Visual Search

Abstract

We propose and evaluate several triplet CNN architectures for measuring the similarity between sketches and photographs, within the context of the sketch based image retrieval (SBIR) task. In contrast to recent fine-grained SBIR work, we study the ability of our networks to generalise across diverse object categories from limited training data, and explore in detail strategies for weight sharing, pre-processing, data augmentation and dimensionality reduction. We exceed the performance of pre-existing techniques on both the Flickr15k category level SBIR benchmark by $18\%$, and the TU-Berlin SBIR benchmark by $\sim 10{\mathcal{T}}_b$, when trained on the 250 category TU-Berlin classification dataset augmented with 25k corresponding photographs harvested from the Internet.

submitted to CVPR2017 on 15Nov16

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Understanding Neural Networks Through Deep Visualization
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Sketch-based 3D Shape Retrieval using Convolutional Neural Networks
• Learning Fine-grained Image Similarity with Deep Ranking
• Deep Image Retrieval: Learning global representations for image search

Cited by in corpus (8)

• The Devil is in the Middle: Exploiting Mid-level Representations for Cross-Domain Instance Matching
• Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval
• Image Hashing by Minimizing Discrete Component-wise Wasserstein Distance
• Image Generation from Sketch Constraint Using Contextual GAN
• Artistic Domain Generalisation Methods are Limited by their Deep Representations
• Deep Shape Matching
• Scalable Visual Attribute Extraction through Hidden Layers of a Residual ConvNet
• LiveSketch: Query Perturbations for Guided Sketch-based Visual Search