SEDD: Scalable and Efficient Dataset Deduplication with GPUs

Youngjun Son, Chaewon Kim, Jaejin Lee

Dataset deduplication is widely recognized as a crucial preprocessing step that enhances data quality and improves the performance of large language models. A commonly used method…