Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement

Qianli Liu, Kaibin Guo, Zicong Hong +5

Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models. Its efficiency depends on the communication and computation latencies of the GPUs, w…