10 papers
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning
Eduardo Sánchez, Rita Berrada, Dan-Mircea Mirea +8
Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code. Linguistic puzzles invert this: the solver must first discover the sys…
Contrastive ESA: Human Evaluation of Multiple Translations at Once
Vilém Zouhar, Roman Grundkiewicz, Sara Rajaee +6
Current human evaluation of machine translation typically assesses single outputs in isolation, a paradigm that suffers from high annotator noise and cost. We introduce Contrastive…
Tiny Aya: Bridging Scale and Multilingual Depth
Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza +23
Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in tran…
Unlocking Reasoning Capability on Machine Translation in Large Language Models
Sara Rajaee, Sebastian Vincent, Alexandre Berard +3
Reasoning-oriented large language models (RLMs) achieve strong gains on tasks such as mathematics and coding by generating explicit intermediate reasoning. However, their impact on…
Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning
Sara Rajaee, Rochelle Choenni, Ekaterina Shutova +1
While the reasoning abilities of large language models (LLMs) continue to advance, it remains unclear how such ability varies across languages in multilingual LLMs and whether diff…
Lost at the Beginning of Reasoning
Baohao Liao, Xinyi Chen, Sara Rajaee +5
Recent advancements in large language models (LLMs) have significantly advanced complex reasoning capabilities, particularly through extended chain-of-thought (CoT) reasoning that…
Local Look-Ahead Guidance via Verifier-in-the-Loop for Automated Theorem Proving
Sara Rajaee, Kumar Pratik, Gabriele Cesa +1
The most promising recent methods for AI reasoning require applying variants of reinforcement learning (RL) either on rolled out trajectories from the LLMs, even for the step-wise…
On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?
Rochelle Choenni, Sara Rajaee, Christof Monz +1
While multilingual language models (MLMs) have been trained on 100+ languages, they are typically only evaluated across a handful of them due to a lack of available test data in mo…
Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models
Sara Rajaee, Christof Monz
Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance…