Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining

Hai Wang, Chenhao Wang, Qifeng Cai +6

Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challengi…