Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts

Weilin Cai, Juyong Jiang, Le Qin +3

Expert parallelism has emerged as a key strategy for distributing the computational workload of sparsely-gated mixture-of-experts (MoE) models across multiple devices, enabling the…