Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly

Abstract

Due to the importance of zero-shot learning, i.e. classifying images where there is a lack of labeled training data, the number of proposed approaches has recently increased steadily. We argue that it is time to take a step back and to analyze the status quo of the area. The purpose of this paper is three-fold. First, given the fact that there is no agreed upon zero-shot learning benchmark, we first define a new benchmark by unifying both the evaluation protocols and data splits of publicly available datasets used for this task. This is an important contribution as published results are often not comparable and sometimes even flawed due to, e.g. pre-training on zero-shot test classes. Moreover, we propose a new zero-shot learning dataset, the Animals with Attributes 2 (AWA2) dataset which we make publicly available both in terms of image features and the images themselves. Second, we compare and analyze a significant number of the state-of-the-art methods in depth, both in the classic zero-shot setting but also in the more realistic generalized zero-shot setting. Finally, we discuss in detail the limitations of the current status of the area which can be taken as a basis for advancing it.

Accepted by TPAMI in July, 2018. We introduce Proposed Split Version 2.0 (Please download it from our project webpage). arXiv admin note: substantial text overlap with arXiv:1703.04394

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Zero-Shot Learning Through Cross-Modal Transfer
• Zero-Shot Learning with Generative Latent Prototype Model

Cited by in corpus (29)

• AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
• Preserving Semantic Relations for Zero-Shot Learning
• Transductive Unbiased Embedding for Zero-Shot Learning
• Discriminative Learning of Latent Features for Zero-Shot Recognition
• Learning to Warm-Start Bayesian Hyperparameter Optimization
• Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects
• Fine-Grained Zero-Shot Learning with DNA as Side Information
• Multi-modal Ensemble Classification for Generalized Zero Shot Learning
• Integrating Semantic Knowledge to Tackle Zero-shot Text Classification
• Relational Generalized Few-Shot Learning
• Episode-based Prototype Generating Network for Zero-Shot Learning
• Learning a Deep Embedding Model for Zero-Shot Learning
• Human Attention in Fine-grained Classification
• Zero-Shot Kernel Learning
• Zero-Shot Semantic Segmentation
• Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations
• Goal-Oriented Gaze Estimation for Zero-Shot Learning
• Adaptive Cross-Modal Few-Shot Learning
• Imagination Based Sample Construction for Zero-Shot Learning
• Learning from Noisy Web Data with Category-level Supervision
• Joint Dictionaries for Zero-Shot Learning
• From Anchor Generation to Distribution Alignment: Learning a Discriminative Embedding Space for Zero-Shot Recognition
• AfriVEC: Word Embedding Models for African Languages. Case Study of Fon and Nobiin
• Zero-shot Transfer Learning for Semantic Parsing
• Global Semantic Consistency for Zero-Shot Learning
• Generalised Zero-Shot Learning with a Classifier Ensemble over Multi-Modal Embedding Spaces
• CIZSL++: Creativity Inspired Generative Zero-Shot Learning
• Learning Single/Multi-Attribute of Object with Symmetry and Group
• Latent Cognizance: What Machine Really Learns