KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

Joao Monteiro, Louis Béthune, Anastasiia Filippova +3

As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negativ…