DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction

Yanqi Zhang, Yuwei Hu, Runyuan Zhao +2

Large language models (LLMs) demonstrate remarkable capabilities but face substantial serving costs due to their high memory demands, with the key-value (KV) cache being a primary…