HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference

Peng Tang, Jiacheng Liu, Xiaofeng Hou +5

The Mixture-of-Experts (MoE) architecture has demonstrated significant advantages in the era of Large Language Models (LLMs), offering enhanced capabilities with reduced inference…