4 papers
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Tomás Brogueira, Marcos Treviso, Miguel Couceiro
Mixture-of-experts (MoE) routers typically apply softmax to the router scores and keep the top-K experts, making every token use exactly K experts. Sparsity-inducing probability ma…
EntmaxKV: Support-Aware Decoding for Entmax Attention
Gonçalo Duarte, Miguel Couceiro, Marcos V. Treviso
Long-context decoding is increasingly limited by KV-cache memory traffic as each generated token attends over a cache whose size grows linearly with context length. Existing sparse…
Which Are the Low-Resource Languages of the Semantic Web?
Ndeye-Emilie Mbengue, Pierre Monnin, Miguel Couceiro +1
Emerging digital technologies are exacerbating the existing divide in Open Access Data (OAD) between high-and low-resource languages, excluding many communities from the global dig…
REFINE-LM: Mitigating Language Model Stereotypes via Reinforcement Learning
Rameez Qureshi, Naïm Es-Sebbani, Luis Galárraga +3
With the introduction of (large) language models, there has been significant concern about the unintended bias such models may inherit from their training data. A number of studies…