SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

Soumia Bouyahiaoui, Manel Kara laouar, Aicha Boutorh +1

Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottlenec…