KV${}^2$: A Self-Refining KV Cache

Johannes Wesch, Danni Liu, Jan Niehues

The memory footprint of the key-value (KV) cache constrains the practical use of long-context models, and it dominates cost when one prefilled context must later serve many differe…