Referenceless Quality Estimation for Natural Language Generation

Abstract

Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.

Accepted as a regular paper to 1st Workshop on Learning to Generate Natural Language (LGNL), Sydney, 10 August 2017

References in corpus (8)

• Improving neural networks by preventing co-adaptation of feature detectors
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
• Neural Language Correction with Character-Based Attention
• Translation Quality Estimation using Recurrent Neural Network
• There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction

Cited by in corpus (11)

• Evaluation of Text Generation: A Survey
• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• Why We Need New Evaluation Metrics for NLG
• The price of debiasing automatic metrics in natural language evaluation
• GO FIGURE: A Meta Evaluation of Factuality in Summarization
• Assessing Crosslingual Discourse Relations in Machine Translation
• Latent Topic Conversational Models
• Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!
• Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
• Evaluating the Morphosyntactic Well-formedness of Generated Texts
• On conducting better validation studies of automatic metrics in natural language generation evaluation