3 citations · 10 across the 21 of their papers we have counts for
6 papers · 1 filter
EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
Shaoting Feng, Yuhan Liu, Hanchen Li +11
Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity…
When to Reason: Semantic Router for vLLM
Chen Wang, Xunzhuo Liu, Yuhan Liu +4
Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also…
LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
Yuhan Liu, Yihua Cheng, Jiayi Yao +8
KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV cach…
AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
Shaoting Feng, Hanchen Li, Kuntai Du +8
Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing L…
PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
Kuntai Du, Bowen Wang, Chen Zhang +9
Besides typical generative applications, like ChatGPT, GitHub Copilot, and Cursor, we observe an emerging trend that LLMs are increasingly used in traditional discriminative tasks,…
Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache
Hanchen Li, Yuhan Liu, Yihua Cheng +2
Across large language model (LLM) applications, we observe an emerging trend for reusing KV caches to save the prefill delays of processing repeated input texts in different LLM in…