4 citations · 10 across the 23 of their papers we have counts for
8 papers · 1 filter
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Mara Finkelstein, Dan Deutsch, Parker Riley +3
As LLMs continue to become more powerful and versatile, human evaluation has quickly become intractable at scale and reliance on automatic metrics has become the norm. Recently, it…
Mitigating Metric Bias in Minimum Bayes Risk Decoding
Geza Kovacs, Daniel Deutsch, Markus Freitag
While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challeng…
Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
Lily H. Zhang, Hamid Dadkhahi, Mara Finkelstein +3
Despite growing interest in incorporating feedback to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of util…
Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data
Zhongtao Liu, Parker Riley, Daniel Deutsch +4
Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. T…
MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
Juraj Juraska, Daniel Deutsch, Mara Finkelstein +1
In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our prim…
Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data
Mara Finkelstein, David Vilar, Markus Freitag
Recent research in neural machine translation (NMT) has shown that training on high-quality machine-generated data can outperform training on human-generated data. This work accomp…
Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms
Firas Trabelsi, David Vilar, Mara Finkelstein +1
Minimum Bayes Risk (MBR) decoding is a powerful decoding strategy widely used for text generation tasks, but its quadratic computational complexity limits its practical application…
Finding Replicable Human Evaluations via Stable Ranking Probability
Parker Riley, Daniel Deutsch, George Foster +3
Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial require…