GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration

Ting Bai, Yue Yu, Le Huang +2

The sparse Mixture-of-Experts (MoE) architecture of large language models (LLMs) confronts an inherent issue of load imbalance arising from the simplistic linear router strategy, w…