The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
arXiv:2303.03915
Abstract
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.
arXiv commentNeurIPS 2022, Datasets and Benchmarks Track
References in corpus (21)
- Scaling Laws for Neural Language Models
- Training Compute-Optimal Large Language Models
- The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Beyond English-Centric Multilingual Machine Translation
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
- Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
- Quantifying Memorization Across Neural Language Models
- Envisioning Communities: A Participatory Approach Towards AI for Social Good
- 1.5 billion words Arabic Corpus
- Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
- Deduplicating Training Data Mitigates Privacy Risks in Language Models
- Data Governance in the Age of Large-Scale Data-Driven Language Technology
- AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages
- Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus
- BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling
- HTLM: Hyper-Text Pre-Training and Prompting of Language Models
- Counterfactual Memorization in Neural Language Models
- A Multilingual Parallel Corpora Collection Effort for Indian Languages
- What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Cited by in corpus (10)
- A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
- Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators
- Artificial Intelligence for Health Message Generation: Theory, Method, and an Empirical Study Using Prompt Engineering
- Designing Heterogeneous LLM Agents for Financial Sentiment Analysis
- Several categories of Large Language Models (LLMs): A Short Survey
- Stronger Together: on the Articulation of Ethical Charters, Legal Tools, and Technical Documentation in ML
- TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese
- Tucano: Advancing Neural Text Generation for Portuguese
- Activations and Gradients Compression for Model-Parallel Training
- Cracks in The Stack: Hidden Vulnerabilities and Licensing Risks in LLM Pre-Training Datasets