Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs

Wentao Ni, Kangqi Zhang, Zhongming Yu +7

As long-context inference becomes central to large language models (LLMs), attention over growing key-value caches emerges as a dominant decoding bottleneck, motivating sparse atte…