MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

Abstract

We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community.

Accepted at LREC 2018

References in corpus (4)

• Teaching Machines to Read and Comprehend
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
• Learning Semantic Script Knowledge with Event Embeddings

Cited by in corpus (11)

• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
• Neural Machine Reading Comprehension: Methods and Trends
• Machine Reading Comprehension: a Literature Review
• What Makes Reading Comprehension Questions Easier?
• Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension
• Complex QA and language models hybrid architectures, Survey
• ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
• A Benchmark Arabic Dataset for Commonsense Explanation
• BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge