DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning

Sikai Bai, Haoxi Li, Jie Zhang +2

Despite the significant breakthrough of Mixture-of-Experts (MoE), the increasing scale of these MoE models presents huge memory and storage challenges. Existing MoE pruning methods…