On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems

Abstract

The ability to compute an accurate reward function is essential for optimising a dialogue policy via reinforcement learning. In real-world applications, using explicit user feedback as the reward signal is often unreliable and costly to collect. This problem can be mitigated if the user's intent is known in advance or data is available to pre-train a task success predictor off-line. In practice neither of these apply for most real world applications. Here we propose an on-line learning framework whereby the dialogue policy is jointly trained alongside the reward model via active learning with a Gaussian process model. This Gaussian process operates on a continuous space dialogue representation generated in an unsupervised fashion using a recurrent neural network encoder-decoder. The experimental results demonstrate that the proposed framework is able to significantly reduce data annotation costs and mitigate noisy user feedback in dialogue policy learning.

arXiv commentAccepted as a long paper in ACL 2016

References in corpus (5)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• A Neural Conversational Model
• PARADISE: A Framework for Evaluating Spoken Dialogue Agents
• Active Learning from Weak and Strong Labelers

Cited by in corpus (28)

• Recent Trends in Deep Learning Based Natural Language Processing
• Deep Reinforcement Learning: An Overview
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• Building a Conversational Agent Overnight with Dialogue Self-Play
• End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
• Latent Intention Dialogue Models
• Recent Advances and Challenges in Task-oriented Dialog System
• Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
• Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
• Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
• RMM: A Recursive Mental Model for Dialog Navigation
• Guided Dialog Policy Learning without Adversarial Learning in the Loop
• Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation
• Teaching Machines to Converse
• VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control
• Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
• GraphDialog: Integrating Graph Knowledge into End-to-End Task-Oriented Dialogue Systems
• Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation
• Counterfactual Off-Policy Training for Neural Response Generation
• Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
• Optimizing Sponsored Search Ranking Strategy by Deep Reinforcement Learning
• A Machine Learning Framework for Data Ingestion in Document Images
• Improving Interaction Quality Estimation with BiLSTMs and the Impact on Dialogue Policy Learning
• NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions
• Integrating User and Agent Models: A Deep Task-Oriented Dialogue System
• Game-Based Video-Context Dialogue