Streaming Attention Approximation via Discrepancy Theory

Ekaterina Kochetkova, Kshiteej Sheth, Insu Han +2

Large language models (LLMs) have achieved impressive success, but their high memory requirements present challenges for long-context token generation. In this paper we study the s…