Non-Autoregressive Neural Dialogue Generation

Abstract

Maximum Mutual information (MMI), which models the bidirectional dependency between responses ($y$) and contexts ($x$), i.e., the forward probability $\log p(y|x)$ and the backward probability $\log p(x|y)$, has been widely used as the objective in the \sts model to address the dull-response issue in open-domain dialog generation. Unfortunately, under the framework of the \sts model, direct decoding from $\log p(y|x)+\log p(x|y)$ is infeasible since the second part (i.e., $p(x|y)$) requires the completion of target generation before it can be computed, and the search space for $y$ is enormous. Empirically, an N-best list is first generated given $p(y|x)$, and $p(x|y)$ is then used to rerank the N-best list, which inevitably results in non-globally-optimal solutions. In this paper, we propose to use non-autoregressive (non-AR) generation model to address this non-global optimality issue. Since target tokens are generated independently in non-AR generation, $p(x|y)$ for each target word can be computed as soon as it's generated, and does not have to wait for the completion of the whole sequence. This naturally resolves the non-global optimal issue in decoding. Experimental results demonstrate that the proposed non-AR strategy produces more diverse, coherent, and appropriate responses, yielding substantive gains in BLEU scores and in human evaluations.

preprint

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• A Neural Conversational Model
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Topic Aware Neural Response Generation
• Towards a Human-like Open-Domain Chatbot
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Adversarial Learning for Neural Dialogue Generation
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
• Mutual Information and Diverse Decoding Improve Neural Machine Translation
• Generative Deep Neural Networks for Dialogue: A Short Review
• Adversarial Evaluation of Dialogue Models
• Learning to Decode for Future Success
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Non-autoregressive Transformer by Position Learning
• Trainable Greedy Decoding for Neural Machine Translation
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• Deep Active Learning for Dialogue Generation
• Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints

Cited by in corpus (6)

• Transformer-based Conditional Variational Autoencoder for Controllable Story Generation
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Defending Against Backdoor Attacks in Natural Language Generation
• OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts
• Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systems
• NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer