1 citations · 1 across the 10 of their papers we have counts for
11 papers
SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops
Ranran Haoran Zhang, Aysa Xuemo Fan, David Munhá Correia +2
Concurrent local LLM serving on unified-memory desktops must preserve memory headroom and output fidelity, which speed-only rankings overlook. We introduce SiliconBench, which eval…
Learning Agent Execution for KV-Cache Management in Agentic Serving
Rui Zhang, Chaeeun Kim, Shaoting Feng +6
Multi-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents. Across these…
Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent
Lingyun Yang, Yuxiao Wang, Shenghao Liang +8
Existing GPU kernel generation benchmarks draw problems from synthetic or curated sources that diverge from deployed workloads. We present Atrex-Bench, a benchmark whose 30 operato…
A Policy-Driven Runtime Layer for Agentic LLM Serving
Rui Zhang, Chaeeun Kim, Liting Hu
Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, role, schemas,…
EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
Shaoting Feng, Yuhan Liu, Hanchen Li +11
Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity…
FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern
Ao Shen, Rui Zhang, Junping Zhao
As large language models (LLMs) continue to scale, multi-node deployment has become a necessity. Consequently, communication has become a critical performance bottleneck. Current i…
Improving Fairness of Large Language Models in Multi-document Summarization
Haoyuan Li, Rui Zhang, Snigdha Chaturvedi
Fairness in multi-document summarization (MDS) is crucial for providing comprehensive views across documents with diverse social attribute values, which can significantly impact de…
eLLM: Elastic Memory Management Framework for Efficient LLM Serving
Jiale Xu, Rui Zhang, Yi Xiong +11
Large Language Models are increasingly being deployed in datacenters. Serving these models requires careful memory management, as their memory usage includes static weights, dynami…
LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
Yi Xiong, Hao Wu, Changxu Shao +6
The expanding context windows in large language models (LLMs) have greatly enhanced their capabilities in various applications, but they also introduce significant challenges in ma…
vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving
Jiale Xu, Rui Zhang, Cong Guo +11
Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. This surge in demand poses significant challenges in optimizing throughp…
GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching
Cong Guo, Rui Zhang, Jiale Xu +8
Large-scale deep neural networks (DNNs), such as large language models (LLMs), have revolutionized the artificial intelligence (AI) field and become increasingly popular. However,…