Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation

Abstract

Non-autoregressive translation (NAT) models remove the dependence on previous target tokens and generate all target tokens in parallel, resulting in significant inference speedup but at the cost of inferior translation accuracy compared to autoregressive translation (AT) models. Considering that AT models have higher accuracy and are easier to train than NAT models, and both of them share the same model configurations, a natural idea to improve the accuracy of NAT models is to transfer a well-trained AT model to an NAT model through fine-tuning. However, since AT and NAT models differ greatly in training strategy, straightforward fine-tuning does not work well. In this work, we introduce curriculum learning into fine-tuning for NAT. Specifically, we design a curriculum in the fine-tuning process to progressively switch the training from autoregressive generation to non-autoregressive generation. Experiments on four benchmark translation datasets show that the proposed method achieves good improvement (more than $1$ BLEU score) over previous NAT baselines in terms of translation accuracy, and greatly speed up (more than $10$ times) the inference process over AT baselines.

arXiv commentAAAI 2020

References in corpus (14)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Convolutional Sequence to Sequence Learning
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• FastSpeech: Fast, Robust and Controllable Text to Speech
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Achieving Human Parity on Automatic Chinese to English News Translation
• Non-Autoregressive Neural Machine Translation
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• On The Power of Curriculum Learning in Training Deep Networks
• Sequence-Level Knowledge Distillation
• Non-Monotonic Sequential Text Generation
• Hint-Based Training for Non-Autoregressive Machine Translation
• Semi-Autoregressive Neural Machine Translation

Cited by in corpus (5)

• Glancing Transformer for Non-Autoregressive Neural Machine Translation
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
• Curriculum Learning with Diversity for Supervised Computer Vision Tasks