Not just a matter of semantics: the relationship between visual similarity and semantic similarity
arXiv:1811.07120
Abstract
Knowledge transfer, zero-shot learning and semantic image retrieval are methods that aim at improving accuracy by utilizing semantic information, e.g. from WordNet. It is assumed that this information can augment or replace missing visual data in the form of labeled training images because semantic similarity correlates with visual similarity. This assumption may seem trivial, but is crucial for the application of such semantic methods. Any violation can cause mispredictions. Thus, it is important to examine the visual-semantic relationship for a certain target problem. In this paper, we use five different semantic and visual similarity measures each to thoroughly analyze the relationship without relying too much on any single definition. We postulate and verify three highly consequential hypotheses on the relationship. Our results show that it indeed exists and that WordNet semantic similarity carries more information about visual similarity than just the knowledge of "different classes look different". They suggest that classification is not the ideal application for semantic methods and that wrong semantic information is much worse than none.
arXiv comment12 pages, 4 figures
References in corpus (6)
- A Survey on Deep Learning in Medical Image Analysis
- Using Information Content to Evaluate Semantic Similarity in a Taxonomy
- Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Do Convolutional Neural Networks Learn Class Hierarchy?
- Hierarchy-based Image Embeddings for Semantic Image Retrieval