HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System

Xiaonan Nie, Pinxue Zhao, Xupeng Miao +2

As giant dense models advance quality but require large amounts of GPU budgets for training, the sparsely gated Mixture-of-Experts (MoE), a kind of conditional computation architec…