Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference

Danil Sivtsov, Aleksandr Katrutsa, Ivan Oseledets

Efficient deployment of a pre-trained LLM to a cluster with multiple servers is a critical step for providing fast responses to users' queries. The recent success of Mixture-of-Exp…