User Response and Sentiment Prediction for Automatic Dialogue Evaluation

Abstract

Automatic evaluation is beneficial for open-domain dialog system development. However, standard word-overlap metrics (BLEU, ROUGE) do not correlate well with human judgements of open-domain dialog systems. In this work we propose to use the sentiment of the next user utterance for turn or dialog level evaluation. Specifically we propose three methods: one that predicts the next sentiment directly, and two others that predict the next user utterance using an utterance or a feedback generator model and then classify its sentiment. Experiments show our model outperforming existing automatic evaluation metrics on both written and spoken open-domain dialogue datasets.

Accepted at EMNLP 2021 Evaluations and Assessments of Neural Conversation Systems Workshop. 2 pages, 1 table

References in corpus (3)

• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• Speech Sentiment and Customer Satisfaction Estimation in Socialbot Conversations