Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior

Abstract

Although neural machine translation models reached high translation quality, the autoregressive nature makes inference difficult to parallelize and leads to high translation latency. Inspired by recent refinement-based approaches, we propose LaNMT, a latent-variable non-autoregressive model with continuous latent variables and deterministic inference procedure. In contrast to existing approaches, we use a deterministic inference algorithm to find the target sequence that maximizes the lowerbound to the log-probability. During inference, the length of translation automatically adapts itself. Our experiments show that the lowerbound can be greatly increased by running the inference algorithm, resulting in significantly improved translation quality. Our proposed model closes the performance gap between non-autoregressive and autoregressive approaches on ASPEC Ja-En dataset with 8.6x faster decoding. On WMT'14 En-De dataset, our model narrows the gap with autoregressive baseline to 2.0 BLEU points with 12.5x speedup. By decoding multiple initial latent variables in parallel and rescore using a teacher model, the proposed model further brings the gap down to 1.0 BLEU point on WMT'14 En-De task with 6.8x speedup.

arXiv commentThis paper was accepted to AAAI 2020, the copyright is transferred to AAAI

References in corpus (14)

• Auto-Encoding Variational Bayes
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Non-Autoregressive Neural Machine Translation
• Image Transformer
• Improving Variational Inference with Inverse Autoregressive Flow
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Non-Monotonic Sequential Text Generation
• Theory and Experiments on Vector Quantized Autoencoders
• Generative Neural Machine Translation
• Avoiding Latent Variable Collapse With Generative Skip Models
• Insertion-based Decoding with automatically Inferred Generation Order
• Improving Sampling from Generative Autoencoders with Markov Chains

Cited by in corpus (5)

• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Non-Autoregressive Neural Dialogue Generation
• On the Discrepancy between Density Estimation and Sequence Generation
• Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation