SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference

Liangkun Chen, Zijian Wen, Tian Wu +2

The Mixture-of-Experts (MoE) architecture has been widely adopted in large language models (LLMs) to reduce computation cost through model sparsity. Employing speculative decoding…