Read What Matters: Query-Adaptive Quantization for KV Caches

Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1

KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for re…