6 papers
SMOL: Professionally translated parallel data for 115 under-represented languages
Isaac Caswell, Elizabeth Nielsen, Jiaming Luo +23
We open-source SMOL (Set of Maximal Overall Leverage), a suite of training data to unlock machine translation for low-resource languages. SMOL has been translated into 124 (and gro…
MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task
Juraj Juraska, Tobias Domhan, Mara Finkelstein +5
In this paper, we present our submissions to the unified WMT25 Translation Evaluation Shared Task. For the Quality Score Prediction subtask, we create a new generation of MetricX w…
Transforming Wearable Data into Personal Health Insights using Large Language Model Agents
Mike A. Merrill, Akshay Paruchuri, Naghmeh Rezaei +17
Deriving personalized insights from popular wearable trackers requires complex numerical reasoning that challenges standard LLMs, necessitating tool-based approaches like code gene…
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects
Daniel Deutsch, Eleftheria Briakou, Isaac Caswell +14
As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual p…
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Mara Finkelstein, Dan Deutsch, Parker Riley +3
As LLMs continue to become more powerful and versatile, human evaluation has quickly become intractable at scale and reliance on automatic metrics has become the norm. Recently, it…
Mitigating Metric Bias in Minimum Bayes Risk Decoding
Geza Kovacs, Daniel Deutsch, Markus Freitag
While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challeng…