Accelerating Distributed MoE Training and Inference with Lina

Jiamin Li, Yimin Jiang, Yibo Zhu +2

Scaling model parameters improves model quality at the price of high computation overhead. Sparsely activated models, usually in the form of Mixture of Experts (MoE) architecture,…