Towards a Comprehensive Scaling Law of Mixture-of-Experts

Guoliang Zhao, Yuhan Fu, Shuaipeng Li +10

Mixture-of-Experts (MoE) models have become the consensus approach for enabling parameter-efficient scaling and cost-effective deployment in large language models. However, existin…