Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches

Sunjoo Whang, Jungjun Oh, Minsung Kim +5

Long inputs and extended generation increase the storage and access costs of the key-value (KV) cache. Low-bit quantization reduces storage and memory traffic, while query-channel…