Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning

Abstract

Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.

arXiv comment12 pages, 8 figures

References in corpus (8)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• A User Simulator for Task-Completion Dialogues
• Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
• Continuously Learning Neural Dialogue Management
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Deep Reinforcement Learning for Multi-Domain Dialogue Systems
• End-to-end optimization of goal-driven and visually grounded dialogue systems
• Investigation of Language Understanding Impact for Reinforcement Learning Based Dialogue Systems

Cited by in corpus (17)

• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• ConvLab: Multi-Domain End-to-End Dialog System Platform
• Video Captioning via Hierarchical Reinforcement Learning
• Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning
• Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
• Learning End-to-End Goal-Oriented Dialog with Maximal User Task Success and Minimal Human Agent Use
• User Modeling for Task Oriented Dialogues
• Guided Dialog Policy Learning without Adversarial Learning in the Loop
• Deep Active Learning for Dialogue Generation
• Prototypical Q Networks for Automatic Conversational Diagnosis and Few-Shot New Disease Adaption
• Joint System-Wise Optimization for Pipeline Goal-Oriented Dialog System
• Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
• Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning
• DialogAct2Vec: Towards End-to-End Dialogue Agent by Multi-Task Representation Learning
• Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management
• Adversarial Learning of Task-Oriented Neural Dialog Models
• Structured Hierarchical Dialogue Policy with Graph Neural Networks