4 papers
SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
Zhenghao Gan, Yichen Bao, Yifei Liu +3
Efficient LLM inference scheduling is crucial for user experience. However, LLM inferences exhibit remarkable demand uncertainty (with unknown output length beforehand) and hybridi…
S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance
Di Liu, Yifei Liu, Chen Chen +4
With the increasing volumes of Large Language Models (LLMs) and the expanding context lengths, attention computation has become a key performance bottleneck in LLM serving. For fas…
Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering
Mingyan Yang, Guanjie Wang, Manqi Luo +6
LLM agents, which often comprise parallel inference tasks, are commonly adopted to solve real-world problems. When serving such task-parallel LLM agents in shared GPU servers, the…
Efficient Unified Caching for Accelerating Heterogeneous AI Workloads
Tianze Wang, Yifei Liu, Chen Chen +8
Modern AI clusters, which host diverse workloads like data pre-processing, training and inference, often store the large-volume data in cloud storage and employ caching frameworks…