SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention

Hong Yankun, Li Xing, Zhen Hui-Ling +3

For the efficient inference of Large Language Models (LLMs), the effective compression of key-value (KV) cache is essential. Three main types of KV cache compression techniques, na…