MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation

Abstract

Conditional masked language models (CMLM) have shown impressive progress in non-autoregressive machine translation (NAT). They learn the conditional translation model by predicting the random masked subset in the target sentence. Based on the CMLM framework, we introduce Multi-view Subset Regularization (MvSR), a novel regularization method to improve the performance of the NAT model. Specifically, MvSR consists of two parts: (1) \textit{shared mask consistency}: we forward the same target with different mask strategies, and encourage the predictions of shared mask positions to be consistent with each other. (2) \textit{model consistency}, we maintain an exponential moving average of the model weights, and enforce the predictions to be consistent between the average model and the online model. Without changing the CMLM-based architecture, our approach achieves remarkable performance on three public benchmarks with 0.36-1.14 BLEU gains over previous NAT models. Moreover, compared with the stronger Transformer baseline, we reduce the gap to 0.01-0.44 BLEU scores on small datasets (WMT16 RO$\leftrightarrow$EN and IWSLT DE$\to$EN).

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• Language Models are Few-Shot Learners
• Cross-lingual Language Model Pretraining
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• R-Drop: Regularized Dropout for Neural Networks
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Adversarial Training for Large Neural Language Models
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Consistency Regularization for Cross-Lingual Fine-Tuning