12 papers
Dynamically Allocating Evaluation Effort for Model Ranking
Vilém Zouhar, Vilém Zouhar, Julia Kreutzer +6
While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation pr…
JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
Erlis Lushtaku, Bora Kargi, Ali Elganzory +4
LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a spe…
The Culture Funnel: You Can't Align What isn't in the Data
Ananya Sahu, Mehrnaz Mofakhami, Daniel D'Souza +3
Current cultural alignment approaches focus on inference-time interventions, assuming models already contain sufficient cultural knowledge. We argue modern LLM pipelines suffer fro…
Tiny Aya: Bridging Scale and Multilingual Depth
Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza +23
Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in tran…
The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
David Mora, Viraat Aryabumi, Wei-Yin Ko +3
Synthetic data has become a cornerstone for scaling large language models, yet its multilingual use remains bottlenecked by translation-based prompts. This strategy inherits Englis…
Making, not Taking, the Best of N
Ammar Khairi, Daniel D'souza, Marzieh Fadaee +1
Obtaining high-quality generations in modern LLMs has largely been framed as a selection problem: identifying a single winning generation from a diverse pool of N samples, the Best…