8 papers
EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
Shaoting Feng, Yuhan Liu, Hanchen Li +11
Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity…
LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
Yuhan Liu, Yihua Cheng, Jiayi Yao +8
KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV cach…
ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
Xingyu Xiang, Raj Joshi, Yuhan Liu +6
Distributed prefix caching accelerates long-context LLM serving by reusing KV cache entries for common context prefixes. However, KV cache fetches can become a bottleneck when netw…
AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
Shaoting Feng, Hanchen Li, Kuntai Du +8
Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing L…
PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
Kuntai Du, Bowen Wang, Chen Zhang +9
Besides typical generative applications, like ChatGPT, GitHub Copilot, and Cursor, we observe an emerging trend that LLMs are increasingly used in traditional discriminative tasks,…
Fact-checking AI-generated news reports: Can LLMs catch their own lies?
Jiayi Yao, Haibo Sun, Nianwen Xue
In this paper, we evaluate the ability of Large Language Models (LLMs) to assess the veracity of claims in ''news reports'' generated by themselves or other LLMs. Our goal is to de…