A Preliminary Study on the Promises and Challenges of Native Top-$k$ Sparse Attention

Di Xiu, Hongyin Tang, Bolin Rong +4

Large Language Models (LLMs) are increasingly prevalent in the field of long-context modeling, however, their inference computational costs have become a critical bottleneck hinder…