3 papers
cs.CL2026
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5
Current hierarchical attention methods, such as NSA and InfLLMv2, select the top-k relevant key-value (KV) blocks based on coarse attention scores and subsequently apply fine-grain…
cs.LG2026
Sparse Attention as Compact Kernel Regression
Saul Santos, Nuno Gonçalves, Daniel C. McNamee +2
Recent work has revealed a link between self-attention mechanisms in transformers and test-time kernel regression via the Nadaraya-Watson estimator, with standard softmax attention…
cs.LG2026
AdaSplash-2: Faster Differentiable Sparse Attention
Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4
Sparse attention has been proposed as a way to alleviate the quadratic cost of transformers, a central bottleneck in long-context training. A promising line of work is -entmax…