Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests

Hongseung Yu, In Gim, Kyunghan Lee +1

Expert offloading enables large mixture-of-experts (MoE) language models to run locally, but conventional batching stalls the GPU while missing experts are fetched. A popular remed…