MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching

Tairan Xu, Leyang Xue, Zhan Lu +2

This paper presents MoE-Gen, a high-throughput MoE inference system optimized for single-GPU execution. Existing inference systems rely on model-based or continuous batching strate…