Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task

Abstract

We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options. Accomplishing the task entails demonstrating comprehension beyond just recognizing "keywords" (or key-phrases) and their corresponding visual concepts. Instead, it requires an alignment between the representations of the two modalities that achieves a visually-grounded "understanding" of various linguistic elements and their dependencies. This new task also admits an easy-to-compute and well-studied metric: the accuracy in detecting the true target among the decoys. The paper makes several contributions: an effective and extensible mechanism for generating decoys from (human-created) image captions; an instance of applying this mechanism, yielding a large-scale machine comprehension dataset (based on the COCO images and captions) that we make publicly available; human evaluation results on this dataset, informing a performance upper-bound; and several baseline and competitive learning approaches that illustrate the utility of the proposed task and dataset in advancing both image and language comprehension. We also show that, in a multi-task learning setting, the performance on the proposed task is positively correlated with the end-to-end task of image captioning.

11 pages

References in corpus (9)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VQA: Visual Question Answering
• Sequence Level Training with Recurrent Neural Networks
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
• Visual Question Answering: A Survey of Methods and Datasets

Cited by in corpus (2)

• Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
• The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA