Language Model Augmented Relevance Score

Abstract

Although automated metrics are commonly used to evaluate NLG systems, they often correlate poorly with human judgements. Newer metrics such as BERTScore have addressed many weaknesses in prior metrics such as BLEU and ROUGE, which rely on n-gram matching. These newer methods, however, are still limited in that they do not consider the generation context, so they cannot properly reward generated text that is correct but deviates from the given reference. In this paper, we propose Language Model Augmented Relevance Score (MARS), a new context-aware metric for NLG evaluation. MARS leverages off-the-shelf language models, guided by reinforcement learning, to create augmented references that consider both the generation context and available human references, which are then used as additional references to score generated text. Compared with seven existing metrics in three common NLG tasks, MARS not only achieves higher correlation with human reference judgements, but also differentiates well-formed candidates from adversarial samples to a larger degree.

arXiv commentIn ACL 2021

References in corpus (8)

• BERTScore: Evaluating Text Generation with BERT
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• mT5: A massively multilingual pre-trained text-to-text transformer
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
• Text Infilling
• Human-Paraphrased References Improve Neural Machine Translation
• Mitigating Political Bias in Language Models Through Reinforced Calibration