Learning a Deep Embedding Model for Zero-Shot Learning

Abstract

Zero-shot learning (ZSL) models rely on learning a joint embedding space where both textual/semantic description of object classes and visual representation of object images can be projected to for nearest neighbour search. Despite the success of deep neural networks that learn an end-to-end model between text and images in other vision problems such as image captioning, very few deep ZSL model exists and they show little advantage over ZSL models that utilise deep feature representations but do not learn an end-to-end embedding. In this paper we argue that the key to make deep ZSL models succeed is to choose the right embedding space. Instead of embedding into a semantic space or an intermediate space, we propose to use the visual space as the embedding space. This is because that in this space, the subsequent nearest neighbour search would suffer much less from the hubness problem and thus become more effective. This model design also provides a natural mechanism for multiple semantic modalities (e.g., attributes and sentence descriptions) to be fused and optimised jointly in an end-to-end manner. Extensive experiments on four benchmarks show that our model significantly outperforms the existing models. Code is available at https://github.com/lzrobots/DeepEmbeddingModel_ZSL

arXiv commentCVPR2017. Adding GZSL results. Code is available at https://github.com/lzrobots/DeepEmbeddingModel_ZSL

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distributed Representations of Words and Phrases and their Compositionality
• Zero-Shot Learning Through Cross-Modal Transfer
• Zero-Shot Learning by Convex Combination of Semantic Embeddings
• Transductive Multi-view Zero-Shot Learning
• Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly
• Actor-Critic Sequence Training for Image Captioning

Cited by in corpus (10)

• Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
• Online Lifelong Generalized Zero-Shot Learning
• TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
• Zero-Shot Sketch-Image Hashing
• Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs
• Zero-Shot Learning via Category-Specific Visual-Semantic Mapping
• Global Semantic Consistency for Zero-Shot Learning
• CIZSL++: Creativity Inspired Generative Zero-Shot Learning
• Sketch-a-Classifier: Sketch-based Photo Classifier Generation
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries