Hold Onto That Thought: Assessing KV Cache Compression On Reasoning

Minghui Liu, Aadi Palnitkar, Tahseen Rabbani +9

Large language models (LLMs) have demonstrated remarkable performance on long-context tasks, but are often bottlenecked by memory constraints. Namely, the KV cache, which is used t…