Speculating Experts Accelerates Inference for Mixture-of-Experts

Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2

Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token…