1 citations · 1 across the 9 of their papers we have counts for
4 papers · 1 filter
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Tomás Brogueira, Marcos Treviso, Miguel Couceiro
Mixture-of-experts (MoE) routers typically apply softmax to the router scores and keep the top-K experts, making every token use exactly K experts. Sparsity-inducing probability ma…
Length Generalization Needs Proper Regularization
Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1
Length generalization is the ability of sequential models to perform well on context lengths unseen during training. In this work, we show that the challenge of achieving length ge…
EntmaxKV: Support-Aware Decoding for Entmax Attention
Gonçalo Duarte, Miguel Couceiro, Marcos V. Treviso
Long-context decoding is increasingly limited by KV-cache memory traffic as each generated token attends over a cache whose size grows linearly with context length. Existing sparse…
AdaSplash-2: Faster Differentiable Sparse Attention
Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4
Sparse attention has been proposed as a way to alleviate the quadratic cost of transformers, a central bottleneck in long-context training. A promising line of work is -entmax a…