Improving Non-autoregressive Neural Machine Translation with Monolingual Data
arXiv:2005.00932
Abstract
Non-autoregressive (NAR) neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. Under this framework, we leverage large monolingual corpora to improve the NAR model's performance, with the goal of transferring the AR model's generalization ability while preventing overfitting. On top of a strong NAR baseline, our experimental results on the WMT14 En-De and WMT16 En-Ro news translation tasks confirm that monolingual data augmentation consistently improves the performance of the NAR model to approach the teacher AR model's performance, yields comparable or better results than the best non-iterative NAR methods in the literature and helps reduce overfitting in the training process.
arXiv commentPublished in ACL 2020
References in corpus (9)
- Sequence to Sequence Learning with Neural Networks
- Non-Autoregressive Neural Machine Translation
- GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
- Sequence-Level Knowledge Distillation
- Understanding Back-Translation at Scale
- Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
- Imitation Learning for Non-Autoregressive Neural Machine Translation
- FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
- Hint-Based Training for Non-Autoregressive Machine Translation