MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

Yangchengyu Zhou, Zhiping Wu, Runchang Jiang +5

The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenari…