Zero-Shot Sketch-Image Hashing

Abstract

Recent studies show that large-scale sketch-based image retrieval (SBIR) can be efficiently tackled by cross-modal binary representation learning methods, where Hamming distance matching significantly speeds up the process of similarity search. Providing training and test data subjected to a fixed set of pre-defined categories, the cutting-edge SBIR and cross-modal hashing works obtain acceptable retrieval performance. However, most of the existing methods fail when the categories of query sketches have never been seen during training. In this paper, the above problem is briefed as a novel but realistic zero-shot SBIR hashing task. We elaborate the challenges of this special task and accordingly propose a zero-shot sketch-image hashing (ZSIH) model. An end-to-end three-network architecture is built, two of which are treated as the binary encoders. The third network mitigates the sketch-image heterogeneity and enhances the semantic relations among data by utilizing the Kronecker fusion layer and graph convolution, respectively. As an important part of ZSIH, we formulate a generative hashing scheme in reconstructing semantic knowledge representations for zero-shot retrieval. To the best of our knowledge, ZSIH is the first zero-shot hashing work suitable for SBIR and cross-modal search. Comprehensive experiments are conducted on two extended datasets, i.e., Sketchy and TU-Berlin with a novel zero-shot train-test split. The proposed model remarkably outperforms related works.

arXiv commentAccepted as spotlight at CVPR 2018

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Sketch-based 3D Shape Retrieval using Convolutional Neural Networks
• Semantic Autoencoder for Zero-Shot Learning
• Zero-Shot Learning via Semantic Similarity Embedding
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Sketch-a-Net that Beats Humans
• Zero-Shot Hashing via Transferring Supervised Knowledge
• Deep Cross-Modal Hashing
• Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval
• Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learning
• Zero-Shot Recognition using Dual Visual-Semantic Mapping Paths
• Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning
• Learning a Deep Embedding Model for Zero-Shot Learning
• Stochastic Generative Hashing
• Zero-Shot Learning via Joint Latent Similarity Embedding

Cited by in corpus (6)

• A Zero-Shot Sketch-based Inter-Modal Object Retrieval Scheme for Remote Sensing Images
• Progressive Domain-Independent Feature Decomposition Network for Zero-Shot Sketch-Based Image Retrieval
• Hybrid-Attention based Decoupled Metric Learning for Zero-Shot Image Retrieval
• Energy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering
• Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval
• Deep Factorised Inverse-Sketching