BERTScore: Evaluating Text Generation with BERT

Abstract

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

Code available at https://github.com/Tiiiger/bert_score; To appear in ICLR2020

References in corpus (16)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Cross-lingual Language Model Pretraining
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Natural Language Inference over Interaction Space
• Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Simple Applications of BERT for Ad Hoc Document Retrieval
• Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• The price of debiasing automatic metrics in natural language evaluation
• A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks
• Alternative Weighting Schemes for ELMo Embeddings
• Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
• Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization

Cited by in corpus (233)

• Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• Revisiting Few-sample BERT Fine-tuning
• Evaluation of Text Generation: A Survey
• Interactive and Explainable Region-guided Radiology Report Generation
• Towards Interpretable Mental Health Analysis with Large Language Models
• Out of the BLEU: how should we assess quality of the Code Generation models?
• Grammatical Error Correction: A Survey of the State of the Art
• An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Abstractive Text Summarization: State of the Art, Challenges, and Improvements
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Recursively Summarizing Books with Human Feedback
• The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
• Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunities
• Identifying and Mitigating the Security Risks of Generative AI
• Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds
• FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Exploring the State of the Art in Legal QA Systems
• Automated Audio Captioning: An Overview of Recent Progress and New Challenges
• Meta-evaluation of Conversational Search Evaluation Metrics
• Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models
• Educational Question Generation of Children Storybooks via Question Type Distribution Learning and Event-Centric Summarization
• NUBIA: NeUral Based Interchangeability Assessor for Text Generation
• UniTE: Unified Translation Evaluation
• BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text
• Abductive Commonsense Reasoning
• NaturalConv: A Chinese Dialogue Dataset Towards Multi-turn Topic-driven Conversation
• QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric
• CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training
• How Large Language Models are Transforming Machine-Paraphrased Plagiarism
• Explaining and Improving Model Behavior with k Nearest Neighbor Representations
• Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers
• Investigating Entropy for Extractive Document Summarization
• On Feature Normalization and Data Augmentation
• Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims
• QuestEval: Summarization Asks for Fact-based Evaluation
• ConceptEVA: Concept-Based Interactive Exploration and Customization of Document Summaries
• MultiGBS: A multi-layer graph approach to biomedical summarization
• Neural Language Generation: Formulation, Methods, and Evaluation
• Summarization, Simplification, and Generation: The Case of Patents
• Unsupervised Topic Segmentation of Meetings with BERT Embeddings
• TellMeWhy: A Dataset for Answering Why-Questions in Narratives
• A Survey of COVID-19 Misinformation: Datasets, Detection Techniques and Open Issues
• On Grounded Planning for Embodied Tasks with Language Models
• GO FIGURE: A Meta Evaluation of Factuality in Summarization
• Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
• Generation of Highlights from Research Papers Using Pointer-Generator Networks and SciBERT Embeddings
• Detecting Hallucinated Content in Conditional Neural Sequence Generation
• Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks
• Uncertainty-Aware Machine Translation Evaluation
• Estimating Redundancy in Clinical Text
• Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
• A Survey on Document-level Neural Machine Translation: Methods and Evaluation
• Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach
• PONE: A Novel Automatic Evaluation Metric for Open-Domain Generative Dialogue Systems
• Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritage
• USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
• Identifying the Adoption or Rejection of Misinformation Targeting COVID-19 Vaccines in Twitter Discourse
• Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
• Revisiting Round-Trip Translation for Quality Estimation
• SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
• Better than Average: Paired Evaluation of NLP Systems
• Enhancing Factual Consistency of Abstractive Summarization
• Scalable Fact-checking with Human-in-the-Loop
• Same Words, Different Meanings: Semantic Polarization in Broadcast Media Language Forecasts Polarization on Social Media Discourse
• A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination
• Challenges in Domain-Specific Abstractive Summarization and How to Overcome them
• Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles
• In Nomine Function: Naming Functions in Stripped Binaries with Neural Networks
• Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
• A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
• Walert: Putting Conversational Search Knowledge into Action by Building and Evaluating a Large Language Model-Powered Chatbot
• Re-evaluating Word Mover's Distance
• Human-Paraphrased References Improve Neural Machine Translation
• Code-switched inspired losses for generic spoken dialog representations
• The Inherent Adversarial Robustness of Analog In-Memory Computing
• Defending Against Backdoor Attacks in Natural Language Generation
• Securing RAG: A Risk Assessment and Mitigation Framework
• Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets
• FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information
• Unsupervised Editing for Counterfactual Stories
• From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey
• Stage-wise Fine-tuning for Graph-to-Text Generation
• Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples
• GLGE: A New General Language Generation Evaluation Benchmark
• More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering
• Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning
• Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings
• LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training
• Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data
• MS2: Multi-Document Summarization of Medical Studies
• Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
• Large Language Models for Combinatorial Optimization: A Systematic Review
• Summary Level Training of Sentence Rewriting for Abstractive Summarization
• Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
• Controllable Text Simplification with Explicit Paraphrasing
• A critical analysis of metrics used for measuring progress in artificial intelligence
• Automatic Logical Forms improve fidelity in Table-to-Text generation
• Paraphrase Types for Generation and Detection
• Instructor-Worker Large Language Model System for Policy Recommendation: a Case Study on Air Quality Analysis of the January 2025 Los Angeles Wildfires
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• Web Archives Metadata Generation with GPT-4o: Challenges and Insights
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
• A Hybrid Intelligence Method for Argument Mining
• GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems
• Automatic Summarization of Russian Texts: Comparison of Extractive and Abstractive Methods
• On the Trade-off between Redundancy and Local Coherence in Summarization
• Controllable Abstractive Dialogue Summarization with Sketch Supervision
• Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• Fast and Accurate Factual Inconsistency Detection Over Long Documents
• Complex QA and language models hybrid architectures, Survey
• Evaluating the Knowledge Dependency of Questions
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods
• CO-NNECT: A Framework for Revealing Commonsense Knowledge Paths as Explicitations of Implicit Knowledge in Texts
• ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer
• Language Model Augmented Relevance Score
• Counterfactual Story Reasoning and Generation
• Research on Tibetan Tourism Viewpoints information generation system based on LLM
• Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding
• FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation
• Regressive Ensemble for Machine Translation Quality Evaluation
• Adversarial Attack and Defense of Structured Prediction Models
• Sentiment-Aware Measure (SAM) for Evaluating Sentiment Transfer by Machine Translation Systems
• VisualMRC: Machine Reading Comprehension on Document Images
• The ADAIO System at the BEA-2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues
• Preserving Semantics in Textual Adversarial Attacks
• TAPE: Assessing Few-shot Russian Language Understanding
• Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation
• Controlling the Amount of Verbatim Copying in Abstractive Summarization
• Generating similes effortlessly like a Pro: A Style Transfer Approach for Simile Generation
• Neural CRF Model for Sentence Alignment in Text Simplification
• MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
• ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generation
• Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
• Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations
• Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation
• EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering
• EMO-KNOW: A Large Scale Dataset on Emotion and Emotion-cause
• Quantitative Evaluation of Alternative Translations in a Corpus of Highly Dissimilar Finnish Paraphrases
• Is In-hospital Meta-information Useful for Abstractive Discharge Summary Generation?
• Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
• Energy-Based Models with Applications to Speech and Language Processing
• Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
• What is More Likely to Happen Next? Video-and-Language Future Event Prediction
• Modeling Disclosive Transparency in NLP Application Descriptions
• Beyond Search Engines: Can Large Language Models Improve Curriculum Development?
• BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation
• Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization
• MMCoVaR: Multimodal COVID-19 Vaccine Focused Data Repository for Fake News Detection and a Baseline Architecture for Classification
• Assessing Dialogue Systems with Distribution Distances
• DEPLAIN: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification
• Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answering
• Conditional Neural Generation using Sub-Aspect Functions for Extractive News Summarization
• What Have We Achieved on Text Summarization?
• Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
• A Data-centric Framework for Improving Domain-specific Machine Reading Comprehension Datasets
• SHADE: Semantic Hypernym Annotator for Domain-specific Entities -- DnD Domain Use Case
• The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents
• Lost In Translation: Generating Adversarial Examples Robust to Round-Trip Translation
• Creating a silver standard for patent simplification
• Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
• Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Data
• Analyzing Multi-Task Learning for Abstractive Text Summarization
• Does Structure Matter? Leveraging Data-to-Text Generation for Answering Complex Information Needs
• When to Talk: Chatbot Controls the Timing of Talking during Multi-turn Open-domain Dialogue Generation
• The statistical advantage of automatic NLG metrics at the system level
• Grammar Accuracy Evaluation (GAE): Quantifiable Quantitative Evaluation of Machine Translation Models
• Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling
• Fine Grained Human Evaluation for English-to-Chinese Machine Translation: A Case Study on Scientific Text
• Gradient-based Adversarial Attacks against Text Transformers
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning
• SciSummPip: An Unsupervised Scientific Paper Summarization Pipeline
• Cue-word Driven Neural Response Generation with a Shrinking Vocabulary
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• On Learning Text Style Transfer with Direct Rewards
• Towards Neural Language Evaluators
• Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization
• Fine-grained Conversational Decoding via Isotropic and Proximal Search
• Automatic Classification of Human Translation and Machine Translation: A Study from the Perspective of Lexical Diversity
• Facet-Aware Evaluation for Extractive Summarization
• Zero-Shot Controlled Generation with Encoder-Decoder Transformers
• Context Matters in Semantically Controlled Language Generation for Task-oriented Dialogue Systems
• Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding
• Generating clickbait spoilers with an ensemble of large language models
• Automatic Claim Review for Climate Science via Explanation Generation
• Evaluation of Semantic Answer Similarity Metrics
• Protecting Anonymous Speech: A Generative Adversarial Network Methodology for Removing Stylistic Indicators in Text
• What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
• SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis
• Which Kind Is Better in Open-domain Multi-turn Dialog,Hierarchical or Non-hierarchical Models? An Empirical Study
• Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
• $Q^2$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
• Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline
• ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
• QMUL-SDS at SCIVER: Step-by-Step Binary Classification for Scientific Claim Verification
• Toward Better Storylines with Sentence-Level Language Models
• SentPWNet: A Unified Sentence Pair Weighting Network for Task-specific Sentence Embedding
• Hierarchical Summarization for Longform Spoken Dialog
• Speech Summarization using Restricted Self-Attention
• Knowledge Enhanced Sports Game Summarization
• Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models
• Diamonds in the Rough: Generating Fluent Sentences from Early-Stage Drafts for Academic Writing Assistance
• Boosting Summarization with Normalizing Flows and Aggressive Training
• Contrastive Semantic Similarity Learning for Image Captioning Evaluation with Intrinsic Auto-encoder
• Adversarial Attacks on Deep Models for Financial Transaction Records
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
• A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions
• Transforming Multi-Conditioned Generation from Meaning Representation
• Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation
• Cross-lingual Transfer of Abstractive Summarizer to Less-resource Language
• ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems
• IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
• CHIME: Cross-passage Hierarchical Memory Network for Generative Review Question Answering
• A Comprehensive Attempt to Research Statement Generation
• GTAE: Graph-Transformer based Auto-Encoders for Linguistic-Constrained Text Style Transfer
• Comparing Visual Reasoning in Humans and AI
• Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance
• HypoGen: Hyperbole Generation with Commonsense and Counterfactual Knowledge
• Online Back-Parsing for AMR-to-Text Generation
• Joint Text and Label Generation for Spoken Language Understanding
• Unsupervised Paraphrasing with Pretrained Language Models
• Towards Human-Free Automatic Quality Evaluation of German Summarization
• PSentScore: Evaluating Sentiment Polarity in Dialogue Summarization
• Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey