KV Cache Compression Through the Lens of Transform Coding

Hannah Laus, Claudio Mayrink Verdun, Hao Wang +2

The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by re…