Hierarchy-based Image Embeddings for Semantic Image Retrieval

Abstract

Deep neural networks trained for classification have been found to learn powerful image representations, which are also often used for other tasks such as comparing images w.r.t. their visual similarity. However, visual similarity does not imply semantic similarity. In order to learn semantically discriminative features, we propose to map images onto class embeddings whose pair-wise dot products correspond to a measure of semantic similarity between classes. Such an embedding does not only improve image retrieval results, but could also facilitate integrating semantics for other tasks, e.g., novelty detection or few-shot learning. We introduce a deterministic algorithm for computing the class centroids directly based on prior world-knowledge encoded in a hierarchy of classes such as WordNet. Experiments on CIFAR-100, NABirds, and ImageNet show that our learned semantic image embeddings improve the semantic consistency of image retrieval results by a large margin.

Accepted at WACV 2019. Source code: https://github.com/cvjena/semantic-embeddings

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On the difficulty of training Recurrent Neural Networks
• Random Erasing Data Augmentation
• YOLO9000: Better, Faster, Stronger
• Significance of Softmax-based Features in Comparison to Distance Metric Learning-based Features
• Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks
• Learning Structured Semantic Embeddings for Visual Recognition

Cited by in corpus (14)

• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Mixed-Precision Embedding Using a Cache
• MMF: Multi-Task Multi-Structure Fusion for Hierarchical Image Classification
• Not just a matter of semantics: the relationship between visual similarity and semantic similarity
• Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
• HUSE: Hierarchical Universal Semantic Embeddings
• Learning Representations For Images With Hierarchical Labels
• Semantic Hierarchy Preserving Deep Hashing for Large-scale Image Retrieval
• SHREWD: Semantic Hierarchy-based Relational Embeddings for Weakly-supervised Deep Hashing
• Understanding top-down attention using task-oriented ablation design
• United We Learn Better: Harvesting Learning Improvements From Class Hierarchies Across Tasks
• Learning to hash with semantic similarity metrics and empirical KL divergence
• Learning Graph-Based Priors for Generalized Zero-Shot Learning