Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
arXiv:1803.08035
Abstract
We consider the problem of zero-shot recognition: learning a visual classifier for a category with zero training examples, just using the word embedding of the category and its relationship to other categories, which visual data are provided. The key to dealing with the unfamiliar or novel category is to transfer knowledge obtained from familiar classes to describe the unfamiliar class. In this paper, we build upon the recently introduced Graph Convolutional Network (GCN) and propose an approach that uses both semantic embeddings and the categorical relationships to predict the classifiers. Given a learned knowledge graph (KG), our approach takes as input semantic embeddings for each node (representing visual category). After a series of graph convolutions, we predict the visual classifier for each category. During training, the visual classifiers for a few categories are given to learn the GCN parameters. At test time, these filters are used to predict the visual classifiers of unseen categories. We show that our approach is robust to noise in the KG. More importantly, our approach provides significant improvement in performance compared to the current state-of-the-art results (from 2 ~ 3% on some metrics to whopping 20% on a few).
arXiv commentCVPR 2018
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- Deep Residual Learning for Image Recognition
- Going Deeper with Convolutions
- Zero-Shot Learning by Convex Combination of Semantic Embeddings
- Return of the Devil in the Details: Delving Deep into Convolutional Nets
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
- Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions
- Local Similarity-Aware Deep Feature Embedding
- Zero-Shot Learning via Semantic Similarity Embedding
- Zero-Shot Learning -- The Good, the Bad and the Ugly
- An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild
- Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
- FVQA: Fact-based Visual Question Answering
- Unsupervised Learning on Neural Network Outputs: with Application in Zero-shot Learning
- Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning
- Learning a Deep Embedding Model for Zero-Shot Learning
- Zero-Shot Learning via Joint Latent Similarity Embedding