BARTScore: Evaluating Generated Text as Text Generation
arXiv:2106.11520
Abstract
A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
arXiv commentNeurIPS 2021
References in corpus (8)
- Teaching Machines to Read and Comprehend
- Semi-supervised Sequence Learning
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
- Few-Shot Text Generation with Pattern-Exploiting Training
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- ExplainaBoard: An Explainable Leaderboard for NLP
- SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Cited by in corpus (28)
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models
- No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
- Towards Interpretable Mental Health Analysis with Large Language Models
- RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
- Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting
- Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization
- CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning
- AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
- UniTE: Unified Translation Evaluation
- Clinical Insights: A Comprehensive Review of Language Models in Medicine
- Automatic Personalized Impression Generation for PET Reports Using Large Language Models
- How Large Language Models are Transforming Machine-Paraphrased Plagiarism
- USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
- ExplainaBoard: An Explainable Leaderboard for NLP
- Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles
- Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
- A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
- Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation
- Automatic Logical Forms improve fidelity in Table-to-Text generation
- Fast and Accurate Factual Inconsistency Detection Over Long Documents
- Complex QA and language models hybrid architectures, Survey
- Multimedia Generative Script Learning for Task Planning
- Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
- Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation
- What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
- Generating Faithful Text From a Knowledge Graph with Noisy Reference Text
- WebQA: Multihop and Multimodal QA