SMat-Attention: Structured Long-Context Sequence Modeling

Emile Anand, Abdullah Ateyeh, Archer Wang +1

Long-context sequence models face a fundamental tradeoff: softmax attention uses flexible token-level interactions at quadratic cost, whereas linear attention obtains linear-time t…