Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
arXiv:2005.10716
Abstract
Open Domain dialog system evaluation is one of the most important challenges in dialog research. Existing automatic evaluation metrics, such as BLEU are mostly reference-based. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots. However, self-reported user rating suffers from bias and variance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them. Specifically, we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples. Our experiments show that CMADE achieves 89.2% accuracy in the dialog comparison task.
References in corpus (14)
- A Neural Conversational Model
- When Does Label Smoothing Help?
- Towards Efficient Data Valuation Based on the Shapley Value
- Data Shapley: Equitable Valuation of Data for Machine Learning
- RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
- ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
- Topic-based Evaluation for Conversational Bots
- Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
- End-to-End Trainable Non-Collaborative Dialog System
- Semi-Supervised Models via Data Augmentationfor Classifying Interactive Affective Responses
- Gunrock: A Social Bot for Complex and Engaging Long Conversations
- Beyond Turing: Intelligent Agents Centered on the User
- Who's responsible? Jointly quantifying the contribution of the learning algorithm and training data
- MOSS: End-to-End Dialog System Framework with Modular Supervision