LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation

Yiqun Shen, Song Yuan, Zhengze Zhang +3

KV Cache is commonly used to accelerate LLM inference with long contexts, yet its high memory demand drives the need for cache compression. Existing compression methods, however, a…