3 citations · 6 across the 19 of their papers we have counts for
6 papers · 1 filter
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Mara Finkelstein, Dan Deutsch, Parker Riley +3
As LLMs continue to become more powerful and versatile, human evaluation has quickly become intractable at scale and reliance on automatic metrics has become the norm. Recently, it…
Mitigating Metric Bias in Minimum Bayes Risk Decoding
Geza Kovacs, Daniel Deutsch, Markus Freitag
While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challeng…
Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data
Zhongtao Liu, Parker Riley, Daniel Deutsch +4
Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. T…
MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
Juraj Juraska, Daniel Deutsch, Mara Finkelstein +1
In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our prim…
Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy
Brian Thompson, Nitika Mathur, Daniel Deutsch +1
Selecting an automatic metric that best emulates human annotators is often non-trivial, because there is no clear definition of "best emulates." A meta-metric is required to compar…
Finding Replicable Human Evaluations via Stable Ranking Probability
Parker Riley, Daniel Deutsch, George Foster +3
Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial require…