CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory

Abstract

Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. The Collaboration-of-Experts (CoE) approach integrates multiple expert models, improving the accuracy of generated results and offering great potential for precision-critical applications, such as automatic circuit board quality inspection. However, deploying CoE serving systems presents challenges to memory capacity due to the large number of experts required, which can lead to significant performance overhead from frequent expert switching across different memory and storage tiers. We propose CoServe, an efficient CoE model serving system on heterogeneous CPU and GPU with limited memory. CoServe reduces unnecessary expert switching by leveraging expert dependency, a key property of CoE inference. CoServe introduces a dependency-aware request scheduler and dependency-aware expert management for efficient inference. It also introduces an offline profiler to automatically find optimal resource allocation on various processors and devices. In real-world intelligent manufacturing workloads, CoServe achieves 4.5$\times$ to 12$\times$ higher throughput compared to state-of-the-art systems.

arXiv commentIn Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '25)

References in corpus (21)

• Scaling Instruction-Finetuned Language Models
• Code Llama: Open Foundation Models for Code
• Mixtral of Experts
• DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
• Jamba: A Hybrid Transformer-Mamba Language Model
• SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts
• EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
• Fast Inference of Mixture-of-Experts Language Models with Offloading
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
• Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
• Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
• Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models
• Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
• JetMoE: Reaching Llama2 Performance with 0.1M Dollars
• Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
• LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
• CCoE: A Compact and Efficient LLM Framework with Multi-Expert Collaboration for Resource-Limited Settings
• Bench-CoE: a Framework for Collaboration of Experts from Benchmark