AdaSplash: Adaptive Sparse Flash Attention

Nuno Gonçalves, Marcos Treviso, André F. T. Martins

The computational cost of softmax-based attention in transformers limits their applicability to long-context tasks. Adaptive sparsity, of which $\alpha$-entmax attention is an example,…