Referenceless Quality Estimation for Natural Language Generation
arXiv:1708.01759
Abstract
Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.
Accepted as a regular paper to 1st Workshop on Learning to Generate Natural Language (LGNL), Sydney, 10 August 2017
References in corpus (5)
- Improving neural networks by preventing co-adaptation of feature detectors
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Translation Quality Estimation using Recurrent Neural Network
- There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction