OrderMoE: An expert similarity driven distributed edge MoE inference

Xin Yuan, Ning Li, Quan Chen +2

Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inferenc…