SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

Shengkun Tang, Zekun Wang, Bo Zheng +7

Structured pruning and knowledge distillation (KD) are typical techniques for compressing large language models, but it remains unclear how they should be applied at pretraining sc…