Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution

Alessio Devoto, Maximilian Jeblick, Simon Jégou

Memory consumption of the Key-Value (KV) cache represents a major bottleneck for efficient large language model inference. While attention-score-based KV cache pruning shows promis…