DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Abstract

Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. However, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We introduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 96k-question benchmark, a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting). These operations require a much more comprehensive understanding of the content of paragraphs than what was necessary for prior datasets. We apply state-of-the-art methods from both the reading comprehension and semantic parsing literature on this dataset and show that the best systems only achieve 32.7% F1 on our generalized accuracy metric, while expert human performance is 96.0%. We additionally present a new model that combines reading comprehension methods with simple numerical reasoning to achieve 47.0% F1.

References in corpus (5)

• Bidirectional Attention Flow for Machine Comprehension
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension
• emrQA: A Large Corpus for Question Answering on Electronic Medical Records

Cited by in corpus (30)

• Language Models are Few-Shot Learners
• Natural Adversarial Examples
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
• Neural Module Networks for Reasoning over Text
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models
• Towards Understanding and Answering Multi-Sentence Recommendation Questions on Tourism
• Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
• MultiModalQA: Complex Question Answering over Text, Tables and Images
• NT5?! Training T5 to Perform Numerical Reasoning
• TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance
• A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering
• Grid Search Hyperparameter Benchmarking of BERT, ALBERT, and LongFormer on DuoRC
• AR-LSAT: Investigating Analytical Reasoning of Text
• Natural Language QA Approaches using Reasoning with External Knowledge
• SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
• Around the GLOBE: Numerical Aggregation Question-Answering on Heterogeneous Genealogical Knowledge Graphs with Deep Neural Networks
• TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
• Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
• An Empirical Investigation of Contextualized Number Prediction
• Unsupervised Domain Adaptation on Reading Comprehension
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Controllable Open-ended Question Generation with A New Question Type Ontology
• Hopper: Multi-hop Transformer for Spatiotemporal Reasoning
• Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation
• AI Marker-based Large-scale AI Literature Mining
• Web Question Answering with Neurosymbolic Program Synthesis
• Abductive Reasoning as Self-Supervision for Common Sense Question Answering
• Challenge Closed-book Science Exam: A Meta-learning Based Question Answering System