most citedLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

2 citations · 2 across the 10 of their papers we have counts for

collaborators
Showing cs.OSShow all

3 papers · 1 filter

cs.OS2026

Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI

Tian Xia, Hanchen Li, Zhifei Li +5

Modern LLM serving systems increasingly host agentic workloads, whose sessions issue tens of model invocations interleaved with tool calls, accumulating KV cache that can be reused…

cs.OS2025

EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving

Shaoting Feng, Yuhan Liu, Hanchen Li +11

Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity…

cs.OS2025

Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

Hanchen Li, Runyuan He, Qiuyang Mang +8

KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. Thi…