Understanding Knowledge Distillation in Non-autoregressive Machine Translation

Abstract

Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

arXiv commentfix a typo; accepted by ICLR 2020

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Convolutional Sequence to Sequence Learning
• Non-Autoregressive Neural Machine Translation
• Levenshtein Transformer
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Mixture Models for Diverse Machine Translation: Tricks of the Trade
• Syntactically Supervised Transformers for Faster Neural Machine Translation
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation

Cited by in corpus (12)

• Knowledge Distillation: A Survey
• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• Semi-Autoregressive Training Improves Mask-Predict Decoding
• Cascaded Text Generation with Markov Transformers
• LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• On the Discrepancy between Density Estimation and Sequence Generation
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
• Autoregressive Knowledge Distillation through Imitation Learning