Hexa-MoE: Efficient and Heterogeneous-aware Training for Mixture-of-Experts

Shuqing Luo, Jie Peng, Pingzhi Li +2

Mixture-of-Experts (MoE) has emerged as a practical approach to scale up parameters for the Transformer model to achieve better generalization while maintaining a sub-linear increa…