Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations

Ajay Jaiswal, Jianyu Wang, Yixiao Li +6

Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks. However, vanilla SMoEs have issues such as expert redundancy…