Towards Standard Criteria for human evaluation of Chatbots: A Survey

Abstract

Human evaluation is becoming a necessity to test the performance of Chatbots. However, off-the-shelf settings suffer the severe reliability and replication issues partly because of the extremely high diversity of criteria. It is high time to come up with standard criteria and exact definitions. To this end, we conduct a through investigation of 105 papers involving human evaluation for Chatbots. Deriving from this, we propose five standard criteria along with precise definitions.

References in corpus (14)

• Deep Reinforcement Learning for Dialogue Generation
• Advances and Challenges in Conversational Recommender Systems: A Survey
• Towards a Human-like Open-Domain Chatbot
• Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• Recent Advances in Neural Question Generation
• Explicit State Tracking with Semi-Supervision for Neural Dialogue Generation
• What makes a good conversation? How controllable attributes affect human judgments
• Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks
• Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement
• NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
• Diversifying Dialogue Generation with Non-Conversational Text
• Generating Dialogue Responses from a Semantic Latent Space