7 papers
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
On the Limits of Model Merging for Multilinguality in Pre-Training
Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2
Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this w…
How Important is `Perfect' English for Machine Translation Prompts?
Patrícia Schmidtová, Niyati Bafna, Seth Aycock +4
Large language models (LLMs) have achieved top results in recent machine translation evaluations, but they are also known to be sensitive to errors and perturbations in their promp…
LLM-as-a-qualitative-judge: automating error analysis in natural language generation
Nadezhda Chirkova, Tunde Oluwaseyi Ajayi, Seth Aycock +4
Prompting large language models (LLMs) to evaluate generated text, known as LLM-as-a-judge, has become a standard evaluation approach in natural language generation (NLG), but is p…
Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation
Di Wu, Seth Aycock, Christof Monz
Large Language Models (LLMs) demonstrate strong reasoning capabilities for many tasks, often by explicitly decomposing the task via Chain-of-Thought (CoT) reasoning. Recent work on…
Masks and Mimicry: Strategic Obfuscation and Impersonation Attacks on Authorship Verification
Kenneth Alperin, Rohan Leekha, Adaku Uchendu +5
The increasing use of Artificial Intelligence (AI) technologies, such as Large Language Models (LLMs) has led to nontrivial improvements in various tasks, including accurate author…
Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?
Seth Aycock, David Stap, Di Wu +2
Extremely low-resource (XLR) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books. Machi…