Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling

Zailin Ma, Quzhe Huang, Yujun Li +2

Data selection for language model pretraining faces a fundamental tension between quality and diversity. While quality filtering is empirically effective, it often induces diversit…