Topic-based Evaluation for Conversational Bots

Abstract

Dialog evaluation is a challenging problem, especially for non task-oriented dialogs where conversational success is not well-defined. We propose to evaluate dialog quality using topic-based metrics that describe the ability of a conversational bot to sustain coherent and engaging conversations on a topic, and the diversity of topics that a bot can handle. To detect conversation topics per utterance, we adopt Deep Average Networks (DAN) and train a topic classifier on a variety of question and query data categorized into multiple topics. We propose a novel extension to DAN by adding a topic-word attention table that allows the system to jointly capture topic keywords in an utterance and perform topic classification. We compare our proposed topic based metrics with the ratings provided by users and show that our metrics both correlate with and complement human judgment. Our analysis is performed on tens of thousands of real human-bot dialogs from the Alexa Prize competition and highlights user expectations for conversational bots.

10 Pages, 2 figures, 9 tables. NIPS 2017 Conversational AI workshop paper. http://alborz-geramifard.com/workshops/nips17-Conversational-AI/Main.html

References in corpus (6)

• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• A Convolutional Neural Network for Modelling Sentences
• A Persona-Based Neural Conversation Model
• Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
• Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

Cited by in corpus (14)

• Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize
• Abstractive and Extractive Text Summarization using Document Context Vector and Recurrent Neural Networks
• What makes a good conversation? How controllable attributes affect human judgments
• Contextual Language Model Adaptation for Conversational Agents
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
• How to Evaluate Your Dialogue Models: A Review of Approaches
• ConCET: Entity-Aware Topic Classification for Open-Domain Conversational Agents
• Contextual Topic Modeling For Dialog Systems
• Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems
• Building A User-Centric and Content-Driven Socialbot
• Open-domain Topic Identification of Out-of-domain Utterances using Wikipedia
• HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation