Apto: Self-Adaptive CPU-GPU Hybrid MoE Inference on Shared PCs

Zihang Zhong, Jiali Wang, Rong Chen

CPU/GPU hybrid inference enables Mixture-of-Experts (MoE) models to run on memory-constrained PCs by keeping all expert weights in host memory and executing activated experts acros…