ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

Abstract

Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.

ICLR 2020 paper. Project page: http://whyu.me/reclor/

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
• HellaSwag: Can a Machine Really Finish Your Sentence?
• MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension

Cited by in corpus (11)

• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
• What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
• Logiformer: A Two-Branch Graph Transformer Network for Interpretable Logical Reasoning
• DAGN: Discourse-Aware Graph Network for Logical Reasoning
• Optimizing Deeper Transformers on Small Datasets
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms
• Learning by Fixing: Solving Math Word Problems with Weak Supervision
• Commonsense-Focused Dialogues for Response Generation: An Empirical Study
• On Commonsense Cues in BERT for Solving Commonsense Tasks
• REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training