Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization

Yaoxiang Wang, Qingguo Hu, Yucheng Ding +6

Mixture-of-Experts (MoE) has emerged as a promising paradigm for efficiently scaling large language models without a proportional increase in computational cost. However, the stand…