Fast Inference of Mixture-of-Experts Language Models with Offloading

Artyom Eliseev, Denis Mazur

With the widespread adoption of Large Language Models (LLMs), many deep learning practitioners are looking for strategies of running these models more efficiently. One such strateg…