6 papers
TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs
Rong Fu, Yibo Meng, Guangzhen Yao +7
Real-time schedulers must reason about tight deadlines under strict compute budgets. We present TempoNet, a reinforcement learning scheduler that pairs a permutation-invariant Tran…
MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
Zhaoyuan Su, Zeyu Zhang, Tingfeng Lan +4
Efficiently serving large language models (LLMs) under dynamic and bursty workloads remains a key challenge for real-world deployment. Existing serving frameworks and static model…
Straggler Tolerant and Resilient DL Training on Homogeneous GPUs
Zeyu Zhang, Haiying Shen
Despite the popularity of homogeneous GPU-based deep learning (DL) training, the prevalence, causes and impact of stragglers and the effectiveness of existing straggler mitigation…
PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference
Zeyu Zhang, Haiying Shen
The scaling of transformer-based Large Language Models (LLMs) has significantly expanded their context lengths, enabling applications where inputs exceed 100K tokens. Our analysis…
FDC: Fast KV Dimensionality Compression for Efficient LLM Inference
Zeyu Zhang, Haiying Shen
In large-language models, memory constraints in the Key-Value Cache (KVC) pose a challenge during inference. In this work, we propose FDC, a fast KV dimensionality compression syst…
HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
Zeyu Zhang, Haiying Shen, Shay Vargaftik +3
Disaggregated Large Language Model (LLM) inference has gained popularity as it separates the computation-intensive prefill stage from the memory-intensive decode stage, avoiding th…