A Probabilistic Interpretation of KV Cache Eviction

Renato Geh, Alex Chen, Daniel Israel +2

The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible cost to quality. This holds…