1 citations · 1 across the 9 of their papers we have counts for
11 papers
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Tomás Brogueira, Marcos Treviso, Miguel Couceiro
Mixture-of-experts (MoE) routers typically apply softmax to the router scores and keep the top-K experts, making every token use exactly K experts. Sparsity-inducing probability ma…
Length Generalization Needs Proper Regularization
Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1
Length generalization is the ability of sequential models to perform well on context lengths unseen during training. In this work, we show that the challenge of achieving length ge…
EntmaxKV: Support-Aware Decoding for Entmax Attention
Gonçalo Duarte, Miguel Couceiro, Marcos V. Treviso
Long-context decoding is increasingly limited by KV-cache memory traffic as each generated token attends over a cache whose size grows linearly with context length. Existing sparse…
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5
Current hierarchical attention methods, such as NSA and InfLLMv2, select the top-k relevant key-value (KV) blocks based on coarse attention scores and subsequently apply fine-grain…
AdaSplash-2: Faster Differentiable Sparse Attention
Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4
Sparse attention has been proposed as a way to alleviate the quadratic cost of transformers, a central bottleneck in long-context training. A promising line of work is -entmax a…
AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos +19
Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translat…
Long-Context Generalization with Sparse Attention
Pavlo Vasylenko, Hugo Pitorro, André F. T. Martins +1
Transformer-based architectures traditionally employ softmax to compute attention weights, which produces dense distributions over all tokens in a sequence. While effective in many…
LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
Hugo Pitorro, Marcos Treviso
State space models (SSMs), such as Mamba, have emerged as an efficient alternative to transformers for long-context sequence modeling. However, despite their growing adoption, SSMs…
AdaSplash: Adaptive Sparse Flash Attention
Nuno Gonçalves, Marcos Treviso, André F. T. Martins
The computational cost of softmax-based attention in transformers limits their applicability to long-context tasks. Adaptive sparsity, of which -entmax attention is an example,…
How Effective are State Space Models for Machine Translation?
Hugo Pitorro, Pavlo Vasylenko, Marcos Treviso +1
Transformers are the current architecture of choice for NLP, but their attention layers do not scale well to long contexts. Recent works propose to replace attention with linear re…
xTower: A Multilingual LLM for Explaining and Correcting Translation Errors
Marcos Treviso, Nuno M. Guerreiro, Sweta Agrawal +7
While machine translation (MT) systems are achieving increasingly strong performance on benchmarks, they often produce translations with errors and anomalies. Understanding these e…