MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning

Tao Zhang, Ziqian Zeng, Hao Peng +2

Long Chain-of-Thought (CoT) reasoning has significantly advanced the capabilities of Large Language Models (LLMs), but this progress is accompanied by substantial memory and latenc…