Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimension

Yichuan Deng, Sridhar Mahadevan, Zhao Song

Large language models (LLMs) have shown their power in different areas. Attention computation, as an important subroutine of LLMs, has also attracted interests in theory. Recently…