Mixture of Diverse Size Experts

Manxi Sun, Wei Liu, Jian Luan +2

The Sparsely-Activated Mixture-of-Experts (MoE) has gained increasing popularity for scaling up large language models (LLMs) without exploding computational costs. Despite its succ…