SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Hongliang Cao, Xiuyu Li, Siqi Wang +3

Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the c…