Task-Specific Expert Pruning for Sparse Mixture-of-Experts

Tianyu Chen, Shaohan Huang, Yuan Xie +5

The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pre-training and has achieved promising results due to its model capacity. However, with trillions of paramete…