Accelerating MoE Model Inference with Expert Sharding

Oana Balmau, Anne-Marie Kermarrec, Rafael Pires +3

Mixture of experts (MoE) models achieve state-of-the-art results in language modeling but suffer from inefficient hardware utilization due to imbalanced token routing and communica…