BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

Yaoyang Liu, Shuwei Shen, Jincheng Tang +5

Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side…