2 papers
cs.LG2026
MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
Sheng Qiang, Ruiwei Chen, Yinpeng Wu +5
Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost. Because the cache grows…
cs.LG2025
DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
Yanqi Zhang, Yuwei Hu, Runyuan Zhao +2
Large language models (LLMs) demonstrate remarkable capabilities but face substantial serving costs due to their high memory demands, with the key-value (KV) cache being a primary…