E8 Lattice Key Quantization for Low-Bit KV-Cache Compression

Sasho Nedelkoski, Alexander Acker, Soeren Becker +2

Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scalar quantization methods achieve…