Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Abstract

With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.

arXiv commentAccepted at TACL; pre-MIT Press publication version

References in corpus (13)

• CamemBERT: a Tasty French Language Model
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Beyond English-Centric Multilingual Machine Translation
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
• GREEK-BERT: The Greeks visiting Sesame Street
• The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
• A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
• On the Impact of Various Types of Noise on Neural Machine Translation
• AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• Pitfalls in Machine Learning Research: Reexamining the Development Cycle

Cited by in corpus (19)

• On the Opportunities and Risks of Foundation Models
• Multimodal datasets: misogyny, pornography, and malignant stereotypes
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
• Data Governance in the Age of Large-Scale Data-Driven Language Technology
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• Efficient Transformers with Dynamic Token Pooling
• XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
• Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese
• Less can be more: representational vs. stereotypical gender bias in facial expression recognition
• An Open Dataset and Model for Language Identification
• Beyond Arabic: Software for Perso-Arabic Script Manipulation
• Sinhala-English Parallel Word Dictionary Dataset
• Seeking Sinhala Sentiment: Predicting Facebook Reactions of Sinhala Posts
• Experimental Standards for Deep Learning in Natural Language Processing Research
• Quantifying Geospatial in the Common Crawl Corpus
• Training Dynamic based data filtering may not work for NLP datasets
• Translation in the Wild
• Inscriptis -- A Python-based HTML to text conversion library optimized for knowledge extraction from the Web