MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router

Yanyue Xie, Zhi Zhang, Ding Zhou +6

Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model per…