Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
arXiv:2005.06249
Abstract
Machine reading comprehension (MRC) aims to teach machines to read and comprehend human languages, which is a long-standing goal of natural language processing (NLP). With the burst of deep neural networks and the evolution of contextualized language models (CLMs), the research of MRC has experienced two significant breakthroughs. MRC and CLM, as a phenomenon, have a great impact on the NLP community. In this survey, we provide a comprehensive and comparative review on MRC covering overall research topics about 1) the origin and development of MRC and CLM, with a particular focus on the role of CLMs; 2) the impact of MRC and CLM to the NLP community; 3) the definition, datasets, and evaluation of MRC; 4) general MRC architecture and technical methods in the view of two-stage Encoder-Decoder solving architecture from the insights of the cognitive process of humans; 5) previous highlights, emerging topics, and our empirical analysis, among which we especially focus on what works in different periods of MRC researches. We propose a full-view categorization and new taxonomies on these topics. The primary views we have arrived at are that 1) MRC boosts the progress from language processing to understanding; 2) the rapid improvement of MRC systems greatly benefits from the development of CLMs; 3) the theme of MRC is gradually moving from shallow text matching to cognitive reasoning.
arXiv comment51 pages
References in corpus (65)
- Distributed Representations of Words and Phrases and their Compositionality
- Sequence to Sequence Learning with Neural Networks
- Teaching Machines to Read and Comprehend
- Bidirectional Attention Flow for Machine Comprehension
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- MPNet: Masked and Permuted Pre-training for Language Understanding
- REALM: Retrieval-Augmented Language Model Pre-Training
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
- Dynamic Coattention Networks For Question Answering
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Machine Comprehension Using Match-LSTM and Answer Pointer
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Assessing BERT's Syntactic Abilities
- ReasoNet: Learning to Stop Reading in Machine Comprehension
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Understanding the Behaviors of BERT in Ranking
- Quasar: Datasets for Question Answering by Search and Reading
- TinyBERT: Distilling BERT for Natural Language Understanding
- The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks
- MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
- MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
- FlowQA: Grasping Flow in History for Conversational Machine Comprehension
- Embracing data abundance: BookTest Dataset for Reading Comprehension
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
- A Unified MRC Framework for Named Entity Recognition
- Option Comparison Network for Multiple-choice Reading Comprehension
- Entity-Relation Extraction as Multi-Turn Question Answering
- Words or Characters? Fine-grained Gating for Reading Comprehension
- Dynamically Fused Graph Network for Multi-hop Reasoning
- Is Graph Structure Necessary for Multi-hop Question Answering?
- Dual Co-Matching Network for Multi-choice Reading Comprehension
- Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering
- Multi-range Reasoning for Machine Comprehension
- One-shot Learning for Question-Answering in Gaokao History Challenge
- Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions
- Machine Reading Comprehension: a Literature Review
- Explicit Contextual Semantics for Text Comprehension
- Unifying Question Answering, Text Classification, and Regression via Span Extraction
- GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension
- A Survey on Neural Machine Reading Comprehension
- Question Answering is a Format; When is it Useful?
- Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
- From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap
- Frustratingly Easy Natural Question Answering
- LIMIT-BERT : Linguistic Informed Multi-Task BERT
- MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
- Multi-style Generative Reading Comprehension
- Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension
- Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
- Adversarial Domain Adaptation for Machine Reading Comprehension
- Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
- A Framework for Evaluation of Machine Reading Comprehension Gold Standards
- Graph Sequential Network for Reasoning over Sequences
- MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization
- Explicit Sentence Compression for Neural Machine Translation
- Knowledgeable Dialogue Reading Comprehension on Key Turns
Cited by in corpus (14)
- Educational Question Generation of Children Storybooks via Question Type Distribution Learning and Event-Centric Summarization
- A Survey on Explainability in Machine Reading Comprehension
- Multi-tasking Dialogue Comprehension with Discourse Parsing
- Multi-span Style Extraction for Generative Reading Comprehension
- Relation-aware Video Reading Comprehension for Temporal Language Grounding
- Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension
- Dialogue Graph Modeling for Conversational Machine Reading
- A Data-centric Framework for Improving Domain-specific Machine Reading Comprehension Datasets
- MRCBert: A Machine Reading ComprehensionApproach for Unsupervised Summarization
- Advances in Multi-turn Dialogue Comprehension: A Survey
- Multi-grained Evidence Inference for Multi-choice Reading Comprehension
- Composing Answer from Multi-spans for Reading Comprehension
- Multi-turn Dialogue Reading Comprehension with Pivot Turns and Knowledge
- Accurate Word Representations with Universal Visual Guidance