Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration

Tianyi Bai, Ling Yang, Zhen Hao Wong +9

Efficient data selection is crucial to accelerate the pretraining of language model (LMs). While various methods have been proposed to enhance data efficiency, limited research has…