4 papers · 1 filter
AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
Di Liu, Ruitian Wang, Chen Chen +6
As large language models scale to longer contexts, loading the growing KV cache during attention computation becomes a critical bottleneck. Previous work has shown that attention c…
SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
Zhenghao Gan, Yichen Bao, Yifei Liu +3
Efficient LLM inference scheduling is crucial for user experience. However, LLM inferences exhibit remarkable demand uncertainty (with unknown output length beforehand) and hybridi…
S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance
Di Liu, Yifei Liu, Chen Chen +4
With the increasing volumes of Large Language Models (LLMs) and the expanding context lengths, attention computation has become a key performance bottleneck in LLM serving. For fas…
Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
Ao Xu, Han Zhao, Weihao Cui +7
Large language models (LLMs) are increasingly deployed under the Model-as-a-Service (MaaS) paradigm. To meet stringent quality-of-service (QoS) requirements, existing LLM serving s…