NewsQA: A Machine Comprehension Dataset

Abstract

We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text from the corresponding articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. A thorough analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (0.198 in F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available at https://datasets.maluuba.com/NewsQA.

References in corpus (7)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Machine Comprehension Using Match-LSTM and Answer Pointer
• A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
• Multi-Perspective Context Matching for Machine Comprehension
• Embracing data abundance: BookTest Dataset for Reading Comprehension

Cited by in corpus (53)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Learning to Generate Questions by Learning What not to Generate
• A Joint Model for Question Answering and Question Generation
• QuAC : Question Answering in Context
• Making Neural QA as Simple as Possible but not Simpler
• A Comparative Study of Word Embeddings for Reading Comprehension
• Hierarchical Graph Network for Multi-hop Question Answering
• Machine Reading Comprehension: a Literature Review
• Semi-Supervised QA with Generative Domain-Adaptive Nets
• Efficient and Robust Question Answering from Minimal Context over Documents
• QuestEval: Summarization Asks for Fact-based Evaluation
• Dynamic Fusion Networks for Machine Reading Comprehension
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions
• Question Answering through Transfer Learning from Large Fine-grained Supervision Data
• Structural Embedding of Syntactic Trees for Machine Comprehension
• Review Conversational Reading Comprehension
• Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
• Answering Science Exam Questions Using Query Rewriting with Background Knowledge
• OneStop QAMaker: Extract Question-Answer Pairs from Text in a One-Stop Approach
• DramaQA: Character-Centered Video Story Understanding with Hierarchical QA
• DocVQA: A Dataset for VQA on Document Images
• Hierarchical Question Answering for Long Documents
• Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
• A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
• Cross-Lingual Transfer Learning for Question Answering
• Improving Machine Reading Comprehension with Single-choice Decision and Transfer Learning
• Adversarial Domain Adaptation for Machine Reading Comprehension
• Grid Search Hyperparameter Benchmarking of BERT, ALBERT, and LongFormer on DuoRC
• Improving Neural Question Generation using World Knowledge
• Transfer Learning for Multi-lingual Tasks -- a Survey
• JEC-QA: A Legal-Domain Question Answering Dataset
• Large-scale Cloze Test Dataset Created by Teachers
• TWEETQA: A Social Media Focused Question Answering Dataset
• Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity
• Unsupervised Domain Adaptation on Reading Comprehension
• Robustly Optimized and Distilled Training for Natural Language Understanding
• A Multilingual Modeling Method for Span-Extraction Reading Comprehension
• A Study of the Tasks and Models in Machine Reading Comprehension
• Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
• ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data
• Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance
• Tag and Correct: Question aware Open Information Extraction with Two-stage Decoding
• A Coarse to Fine Question Answering System based on Reinforcement Learning
• CHIME: Cross-passage Hierarchical Memory Network for Generative Review Question Answering
• Understanding in Artificial Intelligence
• Frustratingly Poor Performance of Reading Comprehension Models on Non-adversarial Examples
• ODSQA: Open-domain Spoken Question Answering Dataset