Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization

Weilin Wan, Jingtao Han, Weizhong Zhang +2

Scaling laws govern macroscopic resource allocation for LLMs, yet precise architectural configurations for Mixture-of-Experts (MoE) models remain guided by heuristics. Existing MoE…