HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Abstract

Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.

arXiv commentACL 2021. Code & data available at https://github.com/Weixin-Liang/HERALD/

References in corpus (10)

• Understanding Black-box Predictions via Influence Functions
• Snorkel: Rapid Training Data Creation with Weak Supervision
• Towards Efficient Data Valuation Based on the Shapley Value
• Data Shapley: Equitable Valuation of Data for Machine Learning
• Estimating Training Data Influence by Tracing Gradient Descent
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Topic-based Evaluation for Conversational Bots
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• LRTA: A Transparent Neural-Symbolic Reasoning Framework with Modular Supervision for Visual Question Answering