Progressive Depth Up-scaling via Optimal Transport

Mingzi Cao, Xi Wang, Nikolaos Aletras

Scaling Large Language Models (LLMs) yields performance gains but incurs substantial training costs. Depth up-scaling offers training efficiency by adding new layers to pre-trained…