ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension

Abstract

We present a large-scale dataset, ReCoRD, for machine reading comprehension requiring commonsense reasoning. Experiments on this dataset demonstrate that the performance of state-of-the-art MRC systems fall far behind human performance. ReCoRD represents a challenge for future research to bridge the gap between human and machine commonsense reading comprehension. ReCoRD is available at http://nlp.jhu.edu/record.

arXiv comment14 pages

References in corpus (7)

• Bidirectional Attention Flow for Machine Comprehension
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• A Simple Method for Commonsense Reasoning
• Know What You Don't Know: Unanswerable Questions for SQuAD
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Cited by in corpus (60)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
• Evaluation of Retrieval-Augmented Generation: A Survey
• Neural Entity Linking: A Survey of Models Based on Deep Learning
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• KLUE: Korean Language Understanding Evaluation
• Graph Neural Networks for Natural Language Processing: A Survey
• Finetuned Language Models Are Zero-Shot Learners
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
• RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Unlearnable Examples: Making Personal Data Unexploitable
• LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
• Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
• Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
• Pretrained Transformers Improve Out-of-Distribution Robustness
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• A Survey of Knowledge Enhanced Pre-trained Models
• Optimal Subarchitecture Extraction For BERT
• The Effect of Natural Distribution Shift on Question Answering Models
• Natural Language Processing with Commonsense Knowledge: A Survey
• KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs
• PQuAD: A Persian Question Answering Dataset
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• Studying Strategically: Learning to Mask for Closed-book QA
• Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering
• More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering
• A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
• Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
• A Hybrid Neural Network Model for Commonsense Reasoning
• Adversarial Domain Adaptation for Machine Reading Comprehension
• AR-LSAT: Investigating Analytical Reasoning of Text
• Natural Language QA Approaches using Reasoning with External Knowledge
• Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
• A Framework for Evaluation of Machine Reading Comprehension Gold Standards
• Which *BERT? A Survey Organizing Contextualized Encoders
• Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension
• Probing Across Time: What Does RoBERTa Know and When?
• R3: A Reading Comprehension Benchmark Requiring Reasoning Processes
• Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
• Analyzing Multi-Task Learning for Abstractive Text Summarization
• Unsupervised Deep Structured Semantic Models for Commonsense Reasoning
• Enhancing Review Comprehension with Domain-Specific Commonsense
• Incorporating Connections Beyond Knowledge Embeddings: A Plug-and-Play Module to Enhance Commonsense Reasoning in Machine Reading Comprehension
• Muppet: Massive Multi-task Representations with Pre-Finetuning
• Can Edge Probing Tasks Reveal Linguistic Knowledge in QA Models?
• Understanding in Artificial Intelligence
• COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences
• Extract, Integrate, Compete: Towards Verification Style Reading Comprehension