G-KV: Decoding-Time KV Cache Eviction with Global Attention

Mengqi Liao, Lu Wang, Chaoyun Zhang +7

Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compressio…