SGD-KV: Summarization Guided KV Cache Compression

Zeyu Liu, Woomin Song, Xuandi Fu +5

Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techn…