Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data

Xuemiao Zhang, Liangyu Xu, Feiyu Duan +5

Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that…