Embracing data abundance: BookTest Dataset for Reading Comprehension

Abstract

There is a practically unlimited amount of natural language data available. Still, recent work in text comprehension has focused on datasets which are small relative to current computing possibilities. This article is making a case for the community to move to larger data and as a step in that direction it is proposing the BookTest, a new dataset similar to the popular Children's Book Test (CBT), however more than 60 times larger. We show that training on the new data improves the accuracy of our Attention-Sum Reader model on the original CBT test data by a much larger margin than many recent attempts to improve the model architecture. On one version of the dataset our ensemble even exceeds the human baseline provided by Facebook. We then show in our own human study that there is still space for further improvement.

The first two authors contributed equally to this work. Submitted to EACL 2017. Code and dataset are publicly available

References in corpus (10)

• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• Attentive Pooling Networks
• Blocks and Fuel: Frameworks for deep learning
• A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
• A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
• Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering
• Consensus Attention-based Neural Networks for Chinese Reading Comprehension
• Who did What: A Large-Scale Person-Centered Cloze Dataset
• A Parallel-Hierarchical Model for Machine Comprehension on Sparse Data

Cited by in corpus (17)

• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• NewsQA: A Machine Comprehension Dataset
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• A Comparative Study of Word Embeddings for Reading Comprehension
• Neural Machine Reading Comprehension: Methods and Trends
• PQuAD: A Persian Question Answering Dataset
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering
• A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
• BIOMRC: A Dataset for Biomedical Machine Reading Comprehension
• Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension
• Restoring ancient text using deep learning: a case study on Greek epigraphy
• DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension
• Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension
• Harry Potter and the Action Prediction Challenge from Natural Language