Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation

Abstract

Non-autoregressive neural machine translation (NAT) predicts the entire target sequence simultaneously and significantly accelerates inference process. However, NAT discards the dependency information in a sentence, and thus inevitably suffers from the multi-modality problem: the target tokens may be provided by different possible translations, often causing token repetitions or missing. To alleviate this problem, we propose a novel semi-autoregressive model RecoverSAT in this work, which generates a translation as a sequence of segments. The segments are generated simultaneously while each segment is predicted token-by-token. By dynamically determining segment length and deleting repetitive segments, RecoverSAT is capable of recovering from repetitive and missing token errors. Experimental results on three widely-used benchmark datasets show that our proposed model achieves more than 4$\times$ speedup while maintaining comparable performance compared with the corresponding autoregressive model.

This work has been accepted for publication at ACL2020

References in corpus (8)

• Non-Autoregressive Neural Machine Translation
• Levenshtein Transformer
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Syntactically Supervised Transformers for Faster Neural Machine Translation
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Non-autoregressive Transformer by Position Learning
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation

Cited by in corpus (2)

• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation