Efficient Stagewise Pretraining via Progressive Subnetworks

Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu +4

Recent developments in large language models have sparked interest in efficient pretraining methods. Stagewise training approaches to improve efficiency, like gradual stacking and…