Designing Precise and Robust Dialogue Response Evaluators

Abstract

Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models. Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora. We open-source the code and data in https://github.com/ZHAOTING/dialog-processing.

Accepted at ACL 2020

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• The Curious Case of Neural Text Degeneration
• DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
• Unifying Human and Statistical Evaluation for Natural Language Generation
• Machine Translation Evaluation with BERT Regressor
• Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses