7 papers
Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement
Qianli Liu, Kaibin Guo, Zicong Hong +5
Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models. Its efficiency depends on the communication and computation latencies of the GPUs, w…
GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving
Ao Li, Shangpeng Yang, Fahao Chen +3
Large Language Model (LLM)-based agents demonstrate strong reasoning and execution capabilities on complex tasks when guided by structured instructions, commonly referred to as wor…
Internet of Agents: Fundamentals, Applications, and Challenges
Yuntao Wang, Shaolong Guo, Yanghe Pan +6
With the rapid proliferation of large language models and vision-language models, AI agents have evolved from isolated, task-specific systems into autonomous, interactive entities…
Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
Fahao Chen, Jie Wan, Peng Li +2
Federated fine-tuning of Mixture-of-Experts (MoE)-based large language models (LLMs) is challenging due to their massive computational requirements and the resource constraints of…
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
Ruixiao Li, Fahao Chen, Peng Li
Speculative decoding accelerates Large Language Model (LLM) inference by employing a small speculative model (SSM) to generate multiple candidate tokens and verify them using the L…
SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models
Fahao Chen, Peng Li, Tom H. Luan +2
Speculative decoding has been shown as an effective way to accelerate Large Language Model (LLM) inference by using a Small Speculative Model (SSM) to generate candidate tokens in…