Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selection

Hongyi He, Xiao Liu, Zhenghao Lin +6

High-quality pre-training data is crutial for large language models, where quality captures factual reliability and semantic value, and diversity ensures broad coverage and distrib…