Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus
arXiv:2105.05241
Abstract
Recent literature has underscored the importance of dataset documentation work for machine learning, and part of this work involves addressing "documentation debt" for datasets that have been used widely but documented sparsely. This paper aims to help address documentation debt for BookCorpus, a popular text dataset for training large language models. Notably, researchers have used BookCorpus to train OpenAI's GPT-N models and Google's BERT models, even though little to no documentation exists about the dataset's motivation, composition, collection process, etc. We offer a preliminary datasheet that provides key context and information about BookCorpus, highlighting several notable deficiencies. In particular, we find evidence that (1) BookCorpus likely violates copyright restrictions for many books, (2) BookCorpus contains thousands of duplicated books, and (3) BookCorpus exhibits significant skews in genre representation. We also find hints of other potential deficiencies that call for future research, including problematic content, potential skews in religious representation, and lopsided author contributions. While more work remains, this initial effort to provide a datasheet for BookCorpus adds to growing literature that urges more careful and systematic documentation for machine learning datasets.
arXiv commentWorking paper
References in corpus (6)
- Language Models are Few-Shot Learners
- Semantics derived automatically from language corpora contain human-like biases
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Cited by in corpus (7)
- On the Opportunities and Risks of Foundation Models
- Copyright in Generative Deep Learning
- The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
- Data Governance in the Age of Large-Scale Data-Driven Language Technology
- AI-generated stories favour stability over change: homogeneity and cultural stereotyping in narratives generated by gpt-4o-mini
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
- Frequency Effects on Syntactic Rule Learning in Transformers