Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

Abstract

Non-autoregressive translation (NAT) models, which remove the dependence on previous target tokens from the inputs of the decoder, achieve significantly inference speedup but at the cost of inferior accuracy compared to autoregressive translation (AT) models. Previous work shows that the quality of the inputs of the decoder is important and largely impacts the model accuracy. In this paper, we propose two methods to enhance the decoder inputs so as to improve NAT models. The first one directly leverages a phrase table generated by conventional SMT approaches to translate source tokens to target tokens, which are then fed into the decoder as inputs. The second one transforms source-side word embeddings to target-side word embeddings through sentence-level alignment and word-level adversary learning, and then feeds the transformed word embeddings into the decoder as inputs. Experimental results show our method largely outperforms the NAT baseline~\citep{gu2017non} by $5.11$ BLEU scores on WMT14 English-German task and $4.72$ BLEU scores on WMT16 English-Romanian task.

AAAI 2019

References in corpus (12)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Sequence Level Training with Recurrent Neural Networks
• Non-Autoregressive Neural Machine Translation
• Neural Machine Translation in Linear Time
• Word Translation Without Parallel Data
• An Actor-Critic Algorithm for Sequence Prediction
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Sequence-Level Knowledge Distillation
• Analyzing Uncertainty in Neural Machine Translation
• Double Path Networks for Sequence to Sequence Learning
• Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter

Cited by in corpus (7)

• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Fast Image Caption Generation with Position Alignment
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• Language Graph Distillation for Low-Resource Machine Translation
• Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
• Neural Machine Translation: A Review and Survey