Semantic bottleneck for computer vision tasks

Abstract

This paper introduces a novel method for the representation of images that is semantic by nature, addressing the question of computation intelligibility in computer vision tasks. More specifically, our proposition is to introduce what we call a semantic bottleneck in the processing pipeline, which is a crossing point in which the representation of the image is entirely expressed with natural language , while retaining the efficiency of numerical representations. We show that our approach is able to generate semantic representations that give state-of-the-art results on semantic content-based image retrieval and also perform very well on image classification tasks. Intelligibility is evaluated through user centered experiments for failure detection.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Towards A Rigorous Science of Interpretable Machine Learning
• VQA: Visual Question Answering
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• What Does Explainable AI Really Mean? A New Conceptualization of Perspectives
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• The (Un)reliability of saliency methods
• Contrastive Learning for Image Captioning
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Learning to Disambiguate by Asking Discriminative Questions