A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
arXiv:2006.11880
Abstract
Machine Reading Comprehension (MRC) is a challenging Natural Language Processing(NLP) research field with wide real-world applications. The great progress of this field in recent years is mainly due to the emergence of large-scale datasets and deep learning. At present, a lot of MRC models have already surpassed human performance on various benchmark datasets despite the obvious giant gap between existing MRC models and genuine human-level reading comprehension. This shows the need for improving existing datasets, evaluation metrics, and models to move current MRC models toward "real" understanding. To address the current lack of comprehensive survey of existing MRC tasks, evaluation metrics, and datasets, herein, (1) we analyze 57 MRC tasks and datasets and propose a more precise classification method of MRC tasks with 4 different attributes; (2) we summarized 9 evaluation metrics of MRC tasks, 7 attributes and 10 characteristics of MRC datasets; (3) We also discuss key open issues in MRC research and highlighted future research directions. In addition, we have collected, organized, and published our data on the companion website(https://mrc-datasets.github.io/) where MRC researchers could directly access each MRC dataset, papers, baseline projects, and the leaderboard.
arXiv comment68 pages
References in corpus (20)
- Teaching Machines to Read and Comprehend
- Bidirectional Attention Flow for Machine Comprehension
- Recent Trends in Deep Learning Based Natural Language Processing
- VQA: Visual Question Answering
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- Know What You Don't Know: Unanswerable Questions for SQuAD
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- Quasar: Datasets for Question Answering by Search and Reading
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- Multilingual Extractive Reading Comprehension by Runtime Machine Translation
- Embracing data abundance: BookTest Dataset for Reading Comprehension
- Machine Reading Comprehension: a Literature Review
- A Survey on Neural Machine Reading Comprehension
- Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading
- ReviewQA: a relational aspect-based opinion reading dataset
- Building Large Machine Reading-Comprehension Datasets using Paragraph Vectors