Irreducible Curriculum for Language Model Pretraining

Simin Fan, Martin Jaggi

Automatic data selection and curriculum design for training large language models is challenging, with only a few existing methods showing improvements over standard training. Furt…