Fast KV Compaction via Attention Matching

Adam Zweiger, Xinghong Fu, Han Guo +1

Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction…