Curriculum-Guided Layer Scaling for Language Model Pretraining

Karanpartap Singh, Neil Band, Ehsan Adeli

As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core training stage. Motivated by cogni…