AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies

Bo-Wen Zhang, Liangdong Wang, Ye Yuan +24

In recent years, with the rapid application of large language models across various fields, the scale of these models has gradually increased, and the resources required for their…