SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling

Weiqiao Shan, Ruixiang Mao, Yuang Li +10

Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained…