FloE: On-the-Fly MoE Inference on Memory-constrained GPU

Yuxin Zhou, Zheng Li, Jun Zhang +5

With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices. While offloading expert parameters…