Data Schedules and Learning Rate Schedules Should Be Designed Together

Haodong Wen, Kairong Luo, Tingkai Yan +2

Mid-training (MT) is a training phase in LLM training pipelines that continues pretraining (PT) on smaller datasets of higher value. Taking the PT and MT phases as a whole, trainin…