STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules

Ranajoy Sadhukhan, Euijin Hong, Sheng Cao +9

Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures remains expensive and data-h…