Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

Abstract

Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos QA, a large-scale dataset of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's everyday narratives, asking such questions as "what might be the possible reason of ...?", or "what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on Cosmos QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.github.io/cosmos.

arXiv commentEMNLP'2019

References in corpus (9)

• Teaching Machines to Read and Comprehend
• BERTScore: Evaluating Text Generation with BERT
• Bidirectional Attention Flow for Machine Comprehension
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Dual Co-Matching Network for Multi-choice Reading Comprehension
• Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension

Cited by in corpus (4)

• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering
• Looking Beyond Sentence-Level Natural Language Inference for Downstream Tasks
• A Benchmark Arabic Dataset for Commonsense Explanation