Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

Yongqin Zeng, Sicheng Pan, Jiale Wang +4

Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data rem…