3 papers
cs.DC2026
CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
Qiaoling Chen, Zhisheng Ye, Tian Tang +7
Batch inference for agentic workloads stresses the GPU key-value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory cap…
cs.DC2025
Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving
Ziming Liu, Boyu Tian, Guoteng Wang +15
Mixture-of-Experts (MoE) models challenge serving infrastructures with dynamic, sparse expert utilization, causing instability on conventional systems designed for dense architectu…
cs.LG2025
Twilight: Adaptive Attention Sparsity with Hierarchical Top- Pruning
Chaofan Lin, Jiaming Tang, Shuo Yang +6
Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-val…