A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
arXiv:1606.02858
Abstract
Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data. Hermann et al. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language understanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 73.6% and 76.6% on these two datasets, exceeding current state-of-the-art results by 7-10% and approaching what we believe is the ceiling for performance on this task.
ACL 2016, updated results
Cited by in corpus (67)
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Dynamic Coattention Networks For Question Answering
- Machine Comprehension Using Match-LSTM and Answer Pointer
- ReasoNet: Learning to Stop Reading in Machine Comprehension
- Pathologies of Neural Models Make Interpretations Difficult
- Learning Recurrent Span Representations for Extractive Question Answering
- Multi-Perspective Context Matching for Machine Comprehension
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- Reading Wikipedia to Answer Open-Domain Questions
- NewsQA: A Machine Comprehension Dataset
- Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering
- MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
- MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
- Embracing data abundance: BookTest Dataset for Reading Comprehension
- DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
- Words or Characters? Fine-grained Gating for Reading Comprehension
- Neural ranking models for document retrieval
- Stochastic Answer Networks for Machine Reading Comprehension
- Consensus Attention-based Neural Networks for Chinese Reading Comprehension
- Improving Question Answering with External Knowledge
- A Comparative Study of Word Embeddings for Reading Comprehension
- Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions
- Semi-Supervised QA with Generative Domain-Adaptive Nets
- Machine Reading Comprehension: a Literature Review
- A Co-Matching Model for Multi-choice Reading Comprehension
- Subword-augmented Embedding for Cloze Reading Comprehension
- Dynamic Fusion Networks for Machine Reading Comprehension
- Question Answering from Unstructured Text by Retrieval and Comprehension
- Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
- Smarnet: Teaching Machines to Read and Comprehend Like Human
- What Makes Reading Comprehension Questions Easier?
- Separating Answers from Queries for Neural Reading Comprehension
- Ruminating Reader: Reasoning with Gated Multi-Hop Attention
- Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension
- Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification
- Gated End-to-End Memory Networks
- RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
- Dataset for the First Evaluation on Chinese Machine Reading Comprehension
- Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
- Reasoning Over Paragraph Effects in Situations
- Hierarchical Question Answering for Long Documents
- Toward Extractive Summarization of Online Forum Discussions via Hierarchical Attention Networks
- Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
- Broad Context Language Modeling as Reading Comprehension
- Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension
- Satirical News Detection and Analysis using Attention Mechanism and Linguistic Features
- Expanding search in the space of empirical ML
- Robust and Scalable Differentiable Neural Computer for Question Answering
- Enhanced Word Representations for Bridging Anaphora Resolution
- Large-scale Cloze Test Dataset Created by Teachers
- Be Consistent! Improving Procedural Text Comprehension using Label Consistency
- Dependent Gated Reading for Cloze-Style Question Answering
- A Boo(n) for Evaluating Architecture Performance
- Semantic Matching Against a Corpus: New Applications and Methods
- GenNet : Reading Comprehension with Multiple Choice Questions using Generation and Selection model
- Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension
- Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension
- Dependency-based Hybrid Trees for Semantic Parsing
- Human acceptability judgements for extractive sentence compression
- A Deterministic Algorithm for Bridging Anaphora Resolution
- Implicit Argument Prediction as Reading Comprehension
- Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text
- SCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations
- Reasoning-Driven Question-Answering for Natural Language Understanding
- A Coarse to Fine Question Answering System based on Reinforcement Learning
- Competency Problems: On Finding and Removing Artifacts in Language Data
- Learning to Generate Questions with Adaptive Copying Neural Networks