AdaSplash-2: Faster Differentiable Sparse Attention

Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4

Sparse attention has been proposed as a way to alleviate the quadratic cost of transformers, a central bottleneck in long-context training. A promising line of work is $\hat{I}\pm$-entmax…