Visual Referring Expression Recognition: What Do Systems Actually Learn?
arXiv:1805.11818
Abstract
We present an empirical analysis of the state-of-the-art systems for referring expression recognition -- the task of identifying the object in an image referred to by a natural language expression -- with the goal of gaining insight into how these systems reason about language and vision. Surprisingly, we find strong evidence that even sophisticated and linguistically-motivated models for this task may ignore the linguistic structure, instead relying on shallow correlations introduced by unintended biases in the data selection and annotation process. For example, we show that a system trained and tested on the input image can achieve a precision of 71.2% in top-2 predictions. Furthermore, a system that predicts only the object category given the input can achieve a precision of 84.2% in top-2 predictions. These surprisingly positive results for what should be deficient prediction scenarios suggest that careful analysis of what our models are learning -- and further, how our data is constructed -- is critical as we seek to make substantive progress on grounded language tasks.
arXiv commentNAACL2018 short
References in corpus (8)
- Sequence to Sequence Learning with Neural Networks
- Teaching Machines to Read and Comprehend
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VQA: Visual Question Answering
- Simple Baseline for Visual Question Answering
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Exploring Nearest Neighbor Approaches for Image Captioning
- Modeling Context in Referring Expressions
Cited by in corpus (5)
- Graph-Structured Referring Expression Reasoning in The Wild
- Joint Visual Grounding with Language Scene Graphs
- Challenges and Prospects in Vision and Language Research
- Give Me Something to Eat: Referring Expression Comprehension with Commonsense Knowledge
- Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning