Improving Matching Models with Hierarchical Contextualized Representations for Multi-turn Response Selection

Abstract

In this paper, we study context-response matching with pre-trained contextualized representations for multi-turn response selection in retrieval-based chatbots. Existing models, such as Cove and ELMo, are trained with limited context (often a single sentence or paragraph), and may not work well on multi-turn conversations, due to the hierarchical nature, informal language, and domain-specific words. To address the challenges, we propose pre-training hierarchical contextualized representations, including contextual word-level and sentence-level representations, by learning a dialogue generation model from large-scale conversations with a hierarchical encoder-decoder architecture. Then the two levels of representations are blended into the input and output layer of a matching model respectively. Experimental results on two benchmark conversation datasets indicate that the proposed hierarchical contextualized representations can bring significantly and consistently improvement to existing matching models for response selection.

arXiv comment6 pages, 1 figure

References in corpus (15)

• Distributed Representations of Words and Phrases and their Compositionality
• A Neural Conversational Model
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• A large annotated corpus for learning natural language inference
• Deep Reinforcement Learning for Dialogue Generation
• Topic Aware Neural Response Generation
• A Diversity-Promoting Objective Function for Neural Conversation Models
• Adversarial Learning for Neural Dialogue Generation
• An Information Retrieval Approach to Short Text Conversation
• Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• Improved Deep Learning Baselines for Ubuntu Corpus Dialogs
• A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
• From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots
• Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation
• Syntax-based Deep Matching of Short Texts