OjaKV: Context-Aware Online Low-Rank KV Cache Compression

Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri +3

The expanding long-context capabilities of large language models are constrained by a significant memory bottleneck: the key-value (KV) cache required for autoregressive generation…