Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

Matthias Seeger, Zeyu Zhang, Vihang Patil +2

A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive…