Rico Sennrich

Last Translation Benchmark

Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…

Scaling Unsupervised Word Alignment to Documents via Structural Constraints

Michelle Wastl, Jannis Vamvas, Rico Sennrich

Word alignment has traditionally been studied between sentences, but many cross-lingual tasks increasingly require correspondences across full documents. While recent multilingual…

Reinforcement Learning Elicits Contextual Learning of Unseen Language Translation

Hanxu Hu, Zdeněk Šnajdr, Pinzhen Chen +2

Prior work has shown that large language models (LLMs) can translate unseen or low-resource languages by undergoing continued training or even by encoding a grammar book in their c…

DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning

Hanxu Hu, Yuxuan Wang, Maggie Huan +4

Reinforcement learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for eliciting reasoning capabilities in large language models, particularly in mathematics…

Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties

Jannis Vamvas, Ignacio Pérez Prat, Angela Heldstab +3

Recent strategies for low-resource machine translation rely on LLMs to generate synthetic data based on text in higher-resource languages. We revisit this idea for Romansh, a langu…

SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for Token-level Recognition of Semantic Differences Between Related Documents

Michelle Wastl, Jannis Vamvas, Rico Sennrich

Recognizing semantic differences across documents is crucial for text generation evaluation and content alignment, especially in cross-lingual settings. However, as a standalone ta…

QueST: Incentivizing LLMs to Generate Difficult Problems

Hanxu Hu, Xingxing Zhang, Jannis Vamvas +2

Large Language Models have achieved strong performance on reasoning tasks, solving competition-level coding and math problems. However, their scalability is limited by human-labele…

Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks

Kirill Semenov, Rico Sennrich

For multilingual factual knowledge assessment of LLMs, benchmarks such as MLAMA use template translations that do not take into account the grammatical and semantic information of…

Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader

Jannis Vamvas, Ignacio Pérez Prat, Not Battesta Soliva +14

The Romansh language, spoken in Switzerland, has limited resources for machine translation evaluation. In this paper, we present a benchmark for six varieties of Romansh: Rumantsch…

Robust Native Language Identification through Agentic Decomposition

Ahmet Yavuz Uluslu, Tannon Kew, Tilia Ellendorff +2

Large language models (LLMs) often achieve high performance in native language identification (NLI) benchmarks by leveraging superficial contextual clues such as names, locations,…

The Mediomatix Corpus: Parallel Data for Romansh Language Varieties via Comparable Schoolbooks

Zachary Hopton, Jannis Vamvas, Andrin Büchler +3

The five idioms (i.e., varieties) of the Romansh language are largely standardized and are taught in the schools of the respective communities in Switzerland. In this paper, we pre…

Leveraging In-Context Learning for Political Bias Testing of LLMs

Patrick Haller, Jannis Vamvas, Rico Sennrich +1

A growing body of work has been querying LLMs with political questions to evaluate their potential biases. However, this probing method has limited stability, making comparisons be…

20min-XD: A Comparable Corpus of Swiss News Articles

Michelle Wastl, Jannis Vamvas, Selena Calleri +1

We present 20min-XD (20 Minuten cross-lingual document-level), a French-German, document-level comparable corpus of news articles, sourced from the Swiss online news outlet 20 Minu…

Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents

Hanxu Hu, Jannis Vamvas, Rico Sennrich

LLMs have paved the way for truly simple document-level machine translation, but challenges such as omission errors remain. In this paper, we study a simple method for handling doc…

Examining Multilingual Embedding Models Cross-Lingually Through LLM-Generated Adversarial Examples

Andrianos Michail, Simon Clematide, Rico Sennrich

The evaluation of cross-lingual semantic search models is often limited to existing datasets from tasks such as information retrieval and semantic textual similarity. We introduce…

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki +2

We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a postprocessing step that replaces rare subwords with their component subwords. The technique…

Linear-time Minimum Bayes Risk Decoding with Reference Aggregation

Jannis Vamvas, Rico Sennrich

Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations, but is expensive, even if a sampling-based appr…

Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect

Jannis Vamvas, Noëmi Aepli, Rico Sennrich

Creating neural text encoders for written Swiss German is challenging due to a dearth of training data combined with dialectal variation. In this paper, we build on several existin…