Designing Multimodal Datasets for NLP Challenges
arXiv:2105.05999
Abstract
In this paper, we argue that the design and development of multimodal datasets for natural language processing (NLP) challenges should be enhanced in two significant respects: to more broadly represent commonsense semantic inferences; and to better reflect the dynamics of actions and events, through a substantive alignment of textual and visual information. We identify challenges and tasks that are reflective of linguistic and cognitive competencies that humans have when speaking and reasoning, rather than merely the performance of systems on isolated tasks. We introduce the distinction between challenge-based tasks and competence-based performance, and describe a diagnostic dataset, Recipe-to-Video Questions (R2VQ), designed for testing competence-based comprehension over a multimodal recipe collection (http://r2vq.org/). The corpus contains detailed annotation supporting such inferencing tasks and facilitating a rich set of question families that we use to evaluate NLP systems.
References in corpus (11)
- VQA: Visual Question Answering
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
- Simulating Action Dynamics with Neural Process Networks
- Competence-based Curriculum Learning for Neural Machine Translation
- Faithful Multimodal Explanation for Visual Question Answering
- Topological Sort for Sentence Ordering