Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems

Abstract

We developed a machine learning approach that quantifies multiple aspects of the success or values in Customer Service contacts, at anytime during the interaction. Specifically, the value/reward function regarding to the turn-level behaviors across human agents, chatbots and other hybrid dialog systems is characterized by the incremental information and confidence gain between sentences, based on the token-level predictions from a multi-task neural network trained with only weak signals in dialog-level attributes/states. The resulting model, named Value Profiler, serves as a goal-oriented dialog manager that enhances conversations by regulating automated decisions with its reward and state predictions. It supports both real-time monitoring and scalable offline customer experience evaluation, for both bot- and human-handled contacts. We show how it improves Amazon customer service quality in several applications.

References in corpus (11)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• BERTScore: Evaluating Text Generation with BERT
• The Curious Case of Neural Text Degeneration
• Deep Reinforcement Learning for Dialogue Generation
• A Diversity-Promoting Objective Function for Neural Conversation Models
• Exploration by Random Network Distillation
• Neural Text Generation with Unlikelihood Training
• An Actor-Critic Algorithm for Sequence Prediction
• End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient
• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
• Interactive Classification by Asking Informative Questions