CascadeEP: Asynchronous Expert Execution for MoE Prefill under Attention Imbalance

Jin Qin, Tiancheng Hu, Shiyan Wang +12

Mixture-of-experts (MoE) serving commonly deploys data and expert parallelism (DEP): attention replicas run distinct request batches while routed experts are sharded across an expe…