Discovering KV Cache Eviction Policies via LLM-Guided Program Evolution

Pratik Poudel, Yanzhao Wu, Sumit Jha +1

KV cache compression is critical for long-context inference, yet effective eviction policies remain difficult to design: existing prefill-stage methods often rely on hand-crafted s…