OpenAttention: One Percent Is Enough with Tail Proxies

Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim +8

Sparse attention reads only a small subset of the key-value cache, and how to treat the unread tail remains a key challenge. Discarding the tail biases the output, and sampling it…