Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function

Abstract

Task-oriented dialog systems enable users to accomplish tasks using natural language. State-of-the-art systems respond to users in the same way regardless of their personalities, although personalizing dialogues can lead to higher levels of adoption and better user experiences. Building personalized dialog systems is an important, yet challenging endeavor and only a handful of works took on the challenge. Most existing works rely on supervised learning approaches and require laborious and expensive labeled training data for each user profile. Additionally, collecting and labeling data for each user profile is virtually impossible. In this work, we propose a novel framework, P-ToD, to personalize task-oriented dialog systems capable of adapting to a wide range of user profiles in an unsupervised fashion using a zero-shot generalizable reward function. P-ToD uses a pre-trained GPT-2 as a backbone model and works in three phases. Phase one performs task-specific training. Phase two kicks off unsupervised personalization by leveraging the proximal policy optimization algorithm that performs policy gradients guided by the zero-shot generalizable reward function. Our novel reward function can quantify the quality of the generated responses even for unseen profiles. The optional final phase fine-tunes the personalized model using a few labeled training examples. We conduct extensive experimental analysis using the personalized bAbI dialogue benchmark for five tasks and up to 180 diverse user profiles. The experimental results demonstrate that P-ToD, even when it had access to zero labeled examples, outperforms state-of-the-art supervised personalization models and achieves competitive performance on BLEU and ROUGE metrics when compared to a strong fully-supervised GPT-2 baseline

arXiv comment11 pages, 4 tables, 31st ACM International Conference on Information and Knowledge Management (CIKM'22)

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Trust Region Policy Optimization
• A Survey on Dialogue Systems: Recent Advances and New Frontiers
• MPNet: Masked and Permuted Pre-training for Language Understanding
• A Simple Method for Commonsense Reasoning
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• A Simple Language Model for Task-Oriented Dialogue
• Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models
• Assigning personality/identity to a chatting machine for coherent conversation generation
• Unsupervised Paraphrasing via Deep Reinforcement Learning
• SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems
• A Neural Topical Expansion Framework for Unstructured Persona-oriented Dialogue Generation
• A Cooperative Memory Network for Personalized Task-oriented Dialogue Systems with Incomplete User Profiles

Cited by in corpus (4)

• AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topics
• Proactive Prioritization of App Issues via Contrastive Learning
• Toward Open-domain Slot Filling via Self-supervised Co-training
• Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?