6 papers
Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent
Lingyun Yang, Yuxiao Wang, Shenghao Liang +8
The paper introduces Atrex-Bench, a trace-driven GPU kernel benchmark derived from real production inference workloads, and evaluates LLM-generated kernels, revealing a large perfo…
A Policy-Driven Runtime Layer for Agentic LLM Serving
Rui Zhang, Chaeeun Kim, Liting Hu
Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, role, schemas,…
eLLM: Elastic Memory Management Framework for Efficient LLM Serving
Jiale Xu, Rui Zhang, Yi Xiong +11
Large Language Models are increasingly being deployed in datacenters. Serving these models requires careful memory management, as their memory usage includes static weights, dynami…
EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
Shaoting Feng, Yuhan Liu, Hanchen Li +11
Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity…
FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern
Ao Shen, Rui Zhang, Junping Zhao
As large language models (LLMs) continue to scale, multi-node deployment has become a necessity. Consequently, communication has become a critical performance bottleneck. Current i…
Improving Fairness of Large Language Models in Multi-document Summarization
Haoyuan Li, Rui Zhang, Snigdha Chaturvedi
Fairness in multi-document summarization (MDS) is crucial for providing comprehensive views across documents with diverse social attribute values, which can significantly impact de…