Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
arXiv:2109.02938
Abstract
This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems. While this task was previously rendered through expensive and time-consuming human labor, we present this novel task of automatic naturalness evaluation of generated language. By fine-tuning the BERT model, our proposed naturalness evaluation method shows robust results and outperforms the baselines: support vector machines, bi-directional LSTMs, and BLEURT. In addition, the training speed and evaluation performance of naturalness model are improved by transfer learning from quality and informativeness linguistic knowledge.
arXiv commentaccepted to RANLP 2021
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- BERTScore: Evaluating Text Generation with BERT
- Evaluation of Text Generation: A Survey
- The price of debiasing automatic metrics in natural language evaluation
- Machine Translation Evaluation with BERT Regressor
- Referenceless Quality Estimation for Natural Language Generation