Which Kind Is Better in Open-domain Multi-turn Dialog,Hierarchical or Non-hierarchical Models? An Empirical Study

Abstract

Currently, open-domain generative dialog systems have attracted considerable attention in academia and industry. Despite the success of single-turn dialog generation, multi-turn dialog generation is still a big challenge. So far, there are two kinds of models for open-domain multi-turn dialog generation: hierarchical and non-hierarchical models. Recently, some works have shown that the hierarchical models are better than non-hierarchical models under their experimental settings; meanwhile, some works also demonstrate the opposite conclusion. Due to the lack of adequate comparisons, it's not clear which kind of models are better in open-domain multi-turn dialog generation. Thus, in this paper, we will measure systematically nearly all representative hierarchical and non-hierarchical models over the same experimental settings to check which kind is better. Through extensive experiments, we have the following three important conclusions: (1) Nearly all hierarchical models are worse than non-hierarchical models in open-domain multi-turn dialog generation, except for the HRAN model. Through further analysis, the excellent performance of HRAN mainly depends on its word-level attention mechanism; (2) The performance of other hierarchical models will also obtain a great improvement if integrating the word-level attention mechanism into these models. The modified hierarchical models even significantly outperform the non-hierarchical models; (3) The reason why the word-level attention mechanism is so powerful for hierarchical models is because it can leverage context information more effectively, especially the fine-grained information. Besides, we have implemented all of the models and already released the codes.

References in corpus (11)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• BERTScore: Evaluating Text Generation with BERT
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Towards a Human-like Open-Domain Chatbot
• A Diversity-Promoting Objective Function for Neural Conversation Models
• Neural Responding Machine for Short-Text Conversation
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
• Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
• Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings