9 papers
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation
Joseph Attieh, Timothee Mickus, Anne-Laure Ligozat +2
Knowledge distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly evaluated through translation quality…
KD4MT: A Survey of Knowledge Distillation for Machine Translation
Ona de Gibert, Joseph Attieh, Timothee Mickus +2
Knowledge Distillation (KD) as a research area has gained a lot of traction in recent years as a compression tool to address challenges related to ever-larger models in NLP. Remark…
Scaling Low-Resource MT via Synthetic Data Generation with LLMs
Ona de Gibert, Joseph Attieh, Teemu Vahtola +5
We investigate the potential of LLM-generated synthetic data for improving low-resource Machine Translation (MT). Focusing on seven diverse target languages, we construct a documen…
SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes
Raúl Vázquez, Timothee Mickus, Elaine Zosa +15
We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs).…
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models
Hengyu Luo, Zihao Li, Joseph Attieh +12
Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation…
I Have an Attention Bridge to Sell You: Generalization Capabilities of Modular Translation Architectures
Timothee Mickus, Raúl Vázquez, Joseph Attieh
Modularity is a paradigm of machine translation with the potential of bringing forth models that are large at training time and small during inference. Within this field of study,…
MAMMOTH: Massively Multilingual Modular Open Translation @ Helsinki
Timothee Mickus, Stig-Arne Grönroos, Joseph Attieh +7
NLP in the age of monolithic large language models is approaching its limits in terms of size and information that can be handled. The trend goes to modularization, a necessary ste…
Isotropy, Clusters, and Classifiers
Timothee Mickus, Stig-Arne Grönroos, Joseph Attieh
Whether embedding spaces use all their dimensions equally, i.e., whether they are isotropic, has been a recent subject of discussion. Evidence has been accrued both for and against…