A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

Rahul Krishnan, Volker Schulz

The key-value (KV) cache is the dominant memory bottleneck in long-context language model inference. Existing compression methods apply low-rank factorization or quantization indep…