Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management

Abstract

For task-oriented dialog systems, training a Reinforcement Learning (RL) based Dialog Management module suffers from low sample efficiency and slow convergence speed due to the sparse rewards in RL.To solve this problem, many strategies have been proposed to give proper rewards when training RL, but their rewards lack interpretability and cannot accurately estimate the distribution of state-action pairs in real dialogs. In this paper, we propose a multi-level reward modeling approach that factorizes a reward into a three-level hierarchy: domain, act, and slot. Based on inverse adversarial reinforcement learning, our designed reward model can provide more accurate and explainable reward signals for state-action pairs.Extensive evaluations show that our approach can be applied to a wide range of reinforcement learning-based dialog systems and significantly improves both the performance and the speed of convergence.

arXiv comment9 pages

References in corpus (7)

• Categorical Reparameterization with Gumbel-Softmax
• Generative Adversarial Imitation Learning
• Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning
• ConvLab: Multi-Domain End-to-End Dialog System Platform
• Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems
• Guided Dialog Policy Learning without Adversarial Learning in the Loop
• Adversarial Learning of Task-Oriented Neural Dialog Models