Expert Divergence Learning for MoE-based Language Models

Jiaang Li, Haibin Chen, Langming Liu +9

The Mixture-of-Experts (MoE) architecture is a powerful technique for scaling language models, yet it often suffers from expert homogenization, where experts learn redundant functi…