9 papers
Communication-Efficient Verifiable Attention for LLM Inference
Ziqun Chen, Ming Wu, Michael Heinrich +4
Computation integrity of remote large language model (LLM) serving can be questionable. For conventional deep neural networks (DNNs), the existing TEE-shielded DNN partitioning (TS…
PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning
Wei Gao, Peng Sun, Dmitrii Ustiugov +2
Prompt tuning has become a prominent strategy for enhancing the performance of Large Language Models (LLMs) on downstream tasks. Many IT enterprises now offer Prompt-Tuning-as-a-Se…
SOPRAG: Multi-view Graph Experts Retrieval for Industrial Standard Operating Procedures
Liangtao Lin, Zhaomeng Zhu, Tianwei Zhang +1
Standard Operating Procedures (SOPs) are essential for ensuring operational safety and consistency in industrial environments. However, retrieving and following these procedures pr…
CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
Qiaoling Chen, Zhisheng Ye, Tian Tang +7
Batch inference for agentic workloads stresses the GPU key-value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory cap…
ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
Qiaoling Chen, Zijun Liu, Peng Sun +6
Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75\% of the training time. Speculative d…
Semantic-Aware Scheduling for GPU Clusters with Large Language Models
Zerui Wang, Qinghao Hu, Ana Klimovic +4
Deep learning (DL) schedulers are pivotal in optimizing resource allocation in GPU clusters, but operate with a critical limitation: they are largely blind to the semantic context…