Unveiling Super Experts in Mixture-of-Experts Large Language Models

Zunhai Su, Qingyuan Li, Hao Zhang +6

In this study, we report, for the first time, the discovery and systematic investigation of a distinct subset of experts that play a pivotal role in the MoE LLMs' forward inference…