Imitation Learning for Non-Autoregressive Neural Machine Translation

Abstract

Non-autoregressive translation models (NAT) have achieved impressive inference speedup. A potential issue of the existing NAT algorithms, however, is that the decoding is conducted in parallel, without directly considering previous context. In this paper, we propose an imitation learning framework for non-autoregressive machine translation, which still enjoys the fast translation speed but gives comparable translation performance compared to its auto-regressive counterpart. We conduct experiments on the IWSLT16, WMT14 and WMT16 datasets. Our proposed model achieves a significant speedup over the autoregressive models, while keeping the translation quality comparable to the autoregressive models. By sampling sentence length in parallel at inference time, we achieve the performance of 31.85 BLEU on WMT16 Ro$\to$En and 30.68 BLEU on IWSLT16 En$\to$De.

arXiv commentAccepted by ACL 2019. arXiv admin note: text overlap with arXiv:1902.10245, arXiv:1812.09664 by other authors

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Playing Atari with Deep Reinforcement Learning
• Non-Autoregressive Neural Machine Translation
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Sequence-Level Knowledge Distillation
• Non-Autoregressive Machine Translation with Auxiliary Regularization
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Semi-Autoregressive Neural Machine Translation

Cited by in corpus (18)

• Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• Non-autoregressive Transformer by Position Learning
• Fast Image Caption Generation with Position Alignment
• Glancing Transformer for Non-Autoregressive Neural Machine Translation
• Cascaded Text Generation with Markov Transformers
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
• Neural Machine Translation: Challenges, Progress and Future
• Improving Non-autoregressive Generation with Mixup Training
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Improving Non-autoregressive Neural Machine Translation with Monolingual Data
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• Semi-Autoregressive Image Captioning
• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation
• Autoregressive Knowledge Distillation through Imitation Learning