Separating the Wheat from the Chaff with BREAD: An open-source benchmark and metrics to detect redundancy in text

Isaac Caswell, Lisa Wang, Isabel Papadimitriou

Data quality is a problem that perpetually resurfaces throughout the field of NLP, regardless of task, domain, or architecture, and remains especially severe for lower-resource lan…