Survey on Evaluation Methods for Dialogue Systems

Abstract

In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class.

References in corpus (6)

• A Survey on Dialogue Systems: Recent Advances and New Frontiers
• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• Analyzing and Characterizing User Intent in Information-seeking Conversations
• Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings
• Attentive History Selection for Conversational Question Answering
• A Context-aware Attention Network for Interactive Question Answering

Cited by in corpus (23)

• Advances and Challenges in Conversational Recommender Systems: A Survey
• A Survey of Natural Language Generation
• Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunities
• Evaluating Mixed-initiative Conversational Search Systems via User Simulation
• Meta-evaluation of Conversational Search Evaluation Metrics
• Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
• Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
• A Survey on Dialogue Management in Human-Robot Interaction
• Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search
• Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey
• Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems
• Understanding User Satisfaction with Task-oriented Dialogue Systems
• User Simulation for Evaluating Information Access Systems
• A Systematic Literature Review on Persuasive Technology at the Workplace
• Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
• Leveraging Large Language Models through Natural Language Processing to provide interpretable Machine Learning predictions of mental deterioration in real time
• MutaBot: A Mutation Testing Approach for Chatbots
• OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models
• A Survey on Recent Advances in Conversational Data Generation
• Open-Domain Dialogue Quality Evaluation: Deriving Nugget-level Scores from Turn-level Scores
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
• Towards socially-competent and culturally-adaptive artificial agents Expressive order, interactional disruptions and recovery strategies
• Towards Objective Evaluation of Socially-Situated Conversational Robots: Assessing Human-Likeness through Multimodal User Behaviors