Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation

Abstract

We introduce the multiresolution recurrent neural network, which extends the sequence-to-sequence framework to model natural language generation as two parallel discrete stochastic processes: a sequence of high-level coarse tokens, and a sequence of natural language tokens. There are many ways to estimate or learn the high-level coarse tokens, but we argue that a simple extraction procedure is sufficient to capture a wealth of high-level discourse semantics. Such procedure allows training the multiresolution recurrent neural network by maximizing the exact joint log-likelihood over both sequences. In contrast to the standard log- likelihood objective w.r.t. natural language tokens (word perplexity), optimizing the joint log-likelihood biases the model towards modeling high-level abstractions. We apply the proposed model to the task of dialogue response generation in two challenging domains: the Ubuntu technical support domain, and Twitter conversations. On Ubuntu, the model outperforms competing approaches by a substantial margin, achieving state-of-the-art results according to both automatic evaluation metrics and a human evaluation study. On Twitter, the model appears to generate more relevant and on-topic responses according to automatic evaluation metrics. Finally, our experiments demonstrate that the proposed model is more adept at overcoming the sparsity of natural language and is better able to capture long-term structure.

21 pages, 2 figures, 10 tables

Cited by in corpus (27)

• Assessing The Factual Accuracy of Generated Text
• Modeling Multi-turn Conversation with Deep Utterance Aggregation
• Ensemble-Based Deep Reinforcement Learning for Chatbots
• Deep Learning Based Chatbot Models
• A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions
• Multi-resolution Multi-task Gaussian Processes
• NAOMI: Non-Autoregressive Multiresolution Sequence Imputation
• Synthetic Test Data Generation Using Recurrent Neural Networks: A Position Paper
• ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation
• Multiresolution Tensor Learning for Efficient and Interpretable Spatial Analysis
• Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems
• DLGNet: A Transformer-based Model for Dialogue Response Generation
• Teaching Machines to Converse
• Data-Space Inversion Using a Recurrent Autoencoder for Time-Series Parameterization
• Dialogue History Matters! Personalized Response Selectionin Multi-turn Retrieval-based Chatbots
• Topic-Aware Multi-turn Dialogue Modeling
• Automatically Learning Data Augmentation Policies for Dialogue Tasks
• Adversarial Bootstrapping for Dialogue Model Training
• Deep Reinforcement Learning for Chatbots Using Clustered Actions and Human-Likeness Rewards
• Modeling Topical Relevance for Multi-Turn Dialogue Generation
• Universal Dependency Parsing from Scratch
• Transformation of Dense and Sparse Text Representations
• On the Possibility of Rewarding Structure Learning Agents: Mutual Information on Linguistic Random Sets
• Sequence Prediction using Spectral RNNs
• Content Word-based Sentence Decoding and Evaluating for Open-domain Neural Response Generation
• Hybrid Supervised Reinforced Model for Dialogue Systems
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses