GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices

Yipin Guo, Arun M George, Siddharth Joshi

Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key-value (KV) cache remains a b…