RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes

Abstract

Understanding and reasoning about cooking recipes is a fruitful research direction towards enabling machines to interpret procedural text. In this work, we introduce RecipeQA, a dataset for multimodal comprehension of cooking recipes. It comprises of approximately 20K instructional recipes with multiple modalities such as titles, descriptions and aligned set of images. With over 36K automatically generated question-answer pairs, we design a set of comprehension and reasoning tasks that require joint understanding of images and text, capturing the temporal flow of events and making sense of procedural knowledge. Our preliminary results indicate that RecipeQA will serve as a challenging test bed and an ideal benchmark for evaluating machine comprehension systems. The data and leaderboard are available at http://hucvl.github.io/recipeqa.

arXiv commentEMNLP 2018

References in corpus (8)

• Teaching Machines to Read and Comprehend
• VQA: Visual Question Answering
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
• Simulating Action Dynamics with Neural Process Networks
• Unsupervised Semantic Parsing of Video Collections
• A Diagram Is Worth A Dozen Images

Cited by in corpus (5)

• Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure
• Learning Household Task Knowledge from WikiHow Descriptions
• Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
• Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning