10 papers
Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Zihan Zhao, Baotong Lu, Shengjie Lin +8
Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent s…
Straggler Tolerant and Resilient DL Training on Homogeneous GPUs
Zeyu Zhang, Haiying Shen
Despite the popularity of homogeneous GPU-based deep learning (DL) training, the prevalence, causes and impact of stragglers and the effectiveness of existing straggler mitigation…
MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
Zhaoyuan Su, Zeyu Zhang, Tingfeng Lan +4
Efficiently serving large language models (LLMs) under dynamic and bursty workloads remains a key challenge for real-world deployment. Existing serving frameworks and static model…
Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
Camille Couturier, Spyros Mastorakis, Haiying Shen +2
Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing leng…
DeepCompile: A Compiler-Driven Approach to Optimizing Distributed Deep Learning Training
Masahiro Tanaka, Du Li, Umesh Chand +3
The rapid growth of deep learning models has increased the demand for efficient distributed training strategies. Fully sharded approaches like ZeRO-3 and FSDP partition model param…
HERA: Hybrid Edge-cloud Resource Allocation for Cost-Efficient AI Agents
Shiyi Liu, Haiying Shen, Shuai Che +2
In the realm of AI, large language models (LLMs) like GPT-4, central to the operation of AI agents, predominantly operate in the cloud, incurring high operational costs. With local…