Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems

Abstract

To train a statistical spoken dialogue system (SDS) it is essential that an accurate method for measuring task success is available. To date training has relied on presenting a task to either simulated or paid users and inferring the dialogue's success by observing whether this presented task was achieved or not. Our aim however is to be able to learn from real users acting under their own volition, in which case it is non-trivial to rate the success as any prior knowledge of the task is simply unavailable. User feedback may be utilised but has been found to be inconsistent. Hence, here we present two neural network models that evaluate a sequence of turn-level features to rate the success of a dialogue. Importantly these models make no use of any prior knowledge of the user's task. The models are trained on dialogues generated by a simulated user and the best model is then used to train a policy on-line which is shown to perform at least as well as a baseline system using prior knowledge of the user's task. We note that the models should also be of interest for evaluating SDS and for monitoring a dialogue in rule-based SDS.

Accepted for publication in INTERSPEECH 2015

References in corpus (3)

• Theano: new features and speed improvements
• A Convolutional Neural Network for Modelling Sentences
• Deep Visual-Semantic Alignments for Generating Image Descriptions

Cited by in corpus (17)

• Recent Trends in Deep Learning Based Natural Language Processing
• A Survey on Dialogue Systems: Recent Advances and New Frontiers
• A Deep Reinforcement Learning Chatbot
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• Learning End-to-End Goal-Oriented Dialog
• Generative Deep Neural Networks for Dialogue: A Short Review
• Latent Intention Dialogue Models
• Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems
• Sentiment Adaptive End-to-End Dialog Systems
• Conditional Generation and Snapshot Learning in Neural Dialogue Systems
• Teaching Machines to Converse
• Adversarial Learning of Task-Oriented Neural Dialog Models
• Reinforcement Learning of Speech Recognition System Based on Policy Gradient and Hypothesis Selection
• An Improved Approach of Intention Discovery with Machine Learning for POMDP-based Dialogue Management
• Improving Interaction Quality Estimation with BiLSTMs and the Impact on Dialogue Policy Learning
• Learning End-to-End Goal-Oriented Dialog with Multiple Answers