In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading

Shuning Lin, Yifan He, Yitong Chen

In today's landscape, Mixture of Experts (MoE) is a crucial architecture that has been used by many of the most advanced models. One of the major challenges of MoE models is that t…