Reformulating KV Cache Eviction Problem for Long-Context LLM Inference

Tho Mai, Joo-Young Kim

Large language models (LLMs) support long-context inference but suffer from substantial memory and runtime overhead due to Key-Value (KV) Cache growth. Existing KV Cache eviction m…