UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training

Keqi Deng, Shaoshi Ling, Ruchao Fan +1

Long-context inference in large language models (LLMs) is bottlenecked by the linear growth of the self-attention key-value (KV) cache. Top-k sparse attention alleviates this by lo…