Relevance-Promoting Language Model for Short-Text Conversation

Abstract

Despite the effectiveness of sequence-to-sequence framework on the task of Short-Text Conversation (STC), the issue of under-exploitation of training data (i.e., the supervision signals from query text is \textit{ignored}) still remains unresolved. Also, the adopted \textit{maximization}-based decoding strategies, inclined to generating the generic responses or responses with repetition, are unsuited to the STC task. In this paper, we propose to formulate the STC task as a language modeling problem and tailor-make a training strategy to adapt a language model for response generation. To enhance generation performance, we design a relevance-promoting transformer language model, which performs additional supervised source attention after the self-attention to increase the importance of informative query tokens in calculating the token-level representation. The model further refines the query representation with relevance clues inferred from its multiple references during training. In testing, we adopt a \textit{randomization-over-maximization} strategy to reduce the generation of generic responses. Experimental results on a large Chinese STC dataset demonstrate the superiority of the proposed model on relevance metrics and diversity metrics.\footnote{Code available at https://ai.tencent.com/ailab/nlp/dialogue/.

AAAI 2020

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Structured Self-attentive Sentence Embedding
• The Curious Case of Neural Text Degeneration
• Language Modeling with Gated Convolutional Networks
• Deep Reinforcement Learning for Dialogue Generation
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• A Diversity-Promoting Objective Function for Neural Conversation Models
• Long Short-Term Memory-Networks for Machine Reading
• Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
• Hierarchical Neural Story Generation
• Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
• Coherent Dialogue with Attention-based Language Models
• Supervised Attentions for Neural Machine Translation
• Response Generation by Context-aware Prototype Editing
• Generating Multiple Diverse Responses with Multi-Mapping and Posterior Mapping Selection