Evaluation of Text Generation: A Survey

Abstract

The paper surveys evaluation methods of natural language generation (NLG) systems that have been developed in the last few years. We group NLG evaluation methods into three categories: (1) human-centric evaluation metrics, (2) automatic metrics that require no training, and (3) machine-learned metrics. For each category, we discuss the progress that has been made and the challenges still being faced, with a focus on the evaluation of recently proposed NLG tasks and neural NLG models. We then present two examples for task-specific NLG evaluations for automatic text summarization and long text generation, and conclude the paper by proposing future research directions.

arXiv comment47 pages (revised version)

References in corpus (52)

• Sequence to Sequence Learning with Neural Networks
• Teaching Machines to Read and Comprehend
• BERTScore: Evaluating Text Generation with BERT
• Building Natural-Language Generation Systems
• Enhanced LSTM for Natural Language Inference
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• An efficient framework for learning sentence representations
• Document Embedding with Paragraph Vectors
• Neural Text Generation with Unlikelihood Training
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• Event Representations for Automated Story Generation with Deep Neural Nets
• Texygen: A Benchmarking Platform for Text Generation Models
• Hierarchical Neural Story Generation
• Learning to Ask: Neural Question Generation for Reading Comprehension
• Fluency-Guided Cross-Lingual Image Captioning
• The Radicalization Risks of GPT-3 and Advanced Neural Language Models
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• Recent Advances in Neural Question Generation
• Language GANs Falling Short
• On Accurate Evaluation of GANs for Language Generation
• Adversarial Text Generation via Feature-Mover's Distance
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing
• Challenges in Data-to-Document Generation
• Neural Text Generation: Past, Present and Beyond
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• Plan-And-Write: Towards Better Automatic Storytelling
• PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking
• Unifying Human and Statistical Evaluation for Natural Language Generation
• Revisiting Challenges in Data-to-Text Generation with Fact Grounding
• A Tutorial on Deep Latent Variable Models of Natural Language
• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
• Few-shot Natural Language Generation for Task-Oriented Dialog
• Mind The Facts: Knowledge-Boosted Coherent Abstractive Text Summarization
• Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview
• A Comprehensive Survey of Multilingual Neural Machine Translation
• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
• Word Mover's Embedding: From Word2Vec to Document Embedding
• Strategies for Structuring Story Generation
• Learning to Evaluate Image Captioning
• Referenceless Quality Estimation for Natural Language Generation
• ELI5: Long Form Question Answering
• Human vs Automatic Metrics: on the Importance of Correlation Design
• Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
• An Automatic Machine Translation Evaluation Metric Based on Dependency Parsing Model
• Discourse-Aware Neural Rewards for Coherent Text Generation
• Learning from Fact-checkers: Analysis and Generation of Fact-checking Language
• Towards a Better Metric for Evaluating Question Generation Systems
• Towards Neural Language Evaluators
• Learning Compact Reward for Image Captioning

Cited by in corpus (31)

• Advances and Challenges in Conversational Recommender Systems: A Survey
• Towards Natural Language Interfaces for Data Visualization: A Survey
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• TellMeWhy: A Dataset for Answering Why-Questions in Narratives
• Summarization, Simplification, and Generation: The Case of Patents
• Neural Language Generation: Formulation, Methods, and Evaluation
• Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
• GO FIGURE: A Meta Evaluation of Factuality in Summarization
• DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
• Code-switched inspired losses for generic spoken dialog representations
• Evaluating Commit Message Generation: To BLEU Or Not To BLEU?
• Multitask Learning for Class-Imbalanced Discourse Classification
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• ChainCQG: Flow-Aware Conversational Question Generation
• Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation
• Creating a silver standard for patent simplification
• IGA : An Intent-Guided Authoring Assistant
• Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
• $Q^2$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
• Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey
• Measuring global properties of neural generative model outputs via generating mathematical objects
• A Conditional Generative Matching Model for Multi-lingual Reply Suggestion
• On the interaction of automatic evaluation and task framing in headline style transfer
• Truth-Conditional Captioning of Time Series Data
• Online Back-Parsing for AMR-to-Text Generation
• Simple or Complex? Complexity-Controllable Question Generation with Soft Templates and Deep Mixture of Experts Model
• Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writing
• Do Encoder Representations of Generative Dialogue Models Encode Sufficient Information about the Task ?