Semi-Autoregressive Neural Machine Translation

Abstract

Existing approaches to neural machine translation are typically autoregressive models. While these models attain state-of-the-art translation quality, they are suffering from low parallelizability and thus slow at decoding long sequences. In this paper, we propose a novel model for fast sequence generation --- the semi-autoregressive Transformer (SAT). The SAT keeps the autoregressive property in global but relieves in local and thus is able to produce multiple successive words in parallel at each time step. Experiments conducted on English-German and Chinese-English translation tasks show that the SAT achieves a good balance between translation quality and decoding speed. On WMT'14 English-German translation, the SAT achieves 5.58$\times$ speedup while maintains 88\% translation quality, significantly better than the previous non-autoregressive methods. When produces two words at each time step, the SAT is almost lossless (only 1\% degeneration in BLEU score).

arXiv commentEMNLP 2018

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• WaveNet: A Generative Model for Raw Audio
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Convolutional Sequence to Sequence Learning
• Non-Autoregressive Neural Machine Translation
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Sequence-Level Knowledge Distillation
• Chunk-Based Bi-Scale Decoder for Neural Machine Translation
• Neural Machine Translation via Binary Code Prediction

Cited by in corpus (10)

• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Cascaded Text Generation with Markov Transformers
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• Improving Non-autoregressive Generation with Mixup Training
• Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Semi-Autoregressive Transformer for Image Captioning
• Sequence Generation: From Both Sides to the Middle
• Neural Approximation of an Auto-Regressive Process through Confidence Guided Sampling