Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System

Abstract

Designing task-oriented dialogue systems is a challenging research topic, since it needs not only to generate utterances fulfilling user requests but also to guarantee the comprehensibility. Many previous works trained end-to-end (E2E) models with supervised learning (SL), however, the bias in annotated system utterances remains as a bottleneck. Reinforcement learning (RL) deals with the problem through using non-differentiable evaluation metrics (e.g., the success rate) as rewards. Nonetheless, existing works with RL showed that the comprehensibility of generated system utterances could be corrupted when improving the performance on fulfilling user requests. In our work, we (1) propose modelling the hierarchical structure between dialogue policy and natural language generator (NLG) with the option framework, called HDNO, where the latent dialogue act is applied to avoid designing specific dialogue act representations; (2) train HDNO via hierarchical reinforcement learning (HRL), as well as suggest the asynchronous updates between dialogue policy and NLG during training to theoretically guarantee their convergence to a local maximizer; and (3) propose using a discriminator modelled with language models as an additional reward to further improve the comprehensibility. We test HDNO on MultiWoz 2.0 and MultiWoz 2.1, the datasets on multi-domain dialogues, in comparison with word-level E2E model trained with RL, LaRL and HDSA, showing improvements on the performance evaluated by automatic evaluation metrics and human evaluation. Finally, we demonstrate the semantic meanings of latent dialogue acts to show the explanability for HDNO.

arXiv commentPublished as a conference paper at ICLR 2021

References in corpus (12)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• A Simple Language Model for Task-Oriented Dialogue
• Why We Need New Evaluation Metrics for NLG
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
• Learning End-to-End Goal-Oriented Dialog
• Hierarchical Text Generation and Planning for Strategic Dialogue
• Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning
• Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context
• Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
• A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts
• Multi-Domain Dialogue Acts and Response Co-Generation
• Retrospective and Prospective Mixture-of-Generators for Task-oriented Dialogue Response Generation

Cited by in corpus (6)

• CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
• A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning
• Causal-aware Safe Policy Improvement for Task-oriented dialogue
• SUMBT+LaRL: Effective Multi-domain End-to-end Neural Task-oriented Dialog System
• DORA: Toward Policy Optimization for Task-oriented Dialogue System with Efficient Context
• Retrieve & Memorize: Dialog Policy Learning with Multi-Action Memory