Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation

Abstract

Non-Autoregressive Neural Machine Translation (NAT) achieves significant decoding speedup through generating target words independently and simultaneously. However, in the context of non-autoregressive translation, the word-level cross-entropy loss cannot model the target-side sequential dependency properly, leading to its weak correlation with the translation quality. As a result, NAT tends to generate influent translations with over-translation and under-translation errors. In this paper, we propose to train NAT to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The bag-of-ngrams training objective is differentiable and can be efficiently calculated, which encourages NAT to capture the target-side sequential dependency and correlates well with the translation quality. We validate our approach on three translation tasks and show that our approach largely outperforms the NAT baseline by about 5.0 BLEU scores on WMT14 En$\leftrightarrow$De and about 2.5 BLEU scores on WMT16 En$\leftrightarrow$Ro.

arXiv commentAAAI 2020

References in corpus (12)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Sequence Level Training with Recurrent Neural Networks
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Syntactically Supervised Transformers for Faster Neural Machine Translation
• Theory and Experiments on Vector Quantized Autoencoders
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Non-autoregressive Transformer by Position Learning
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation

Cited by in corpus (17)

• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• Glancing Transformer for Non-Autoregressive Neural Machine Translation
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Fast Interleaved Bidirectional Sequence Generation
• Improving Non-autoregressive Neural Machine Translation with Monolingual Data
• Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
• Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation
• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP
• Modeling Coverage for Non-Autoregressive Neural Machine Translation
• AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate