Learning Light-Weight Translation Models from Deep Transformer
arXiv:2012.13866
Abstract
Recently, deep models have shown tremendous improvements in neural machine translation (NMT). However, systems of this kind are computationally expensive and memory intensive. In this paper, we take a natural step towards learning strong but light-weight NMT systems. We proposed a novel group-permutation based knowledge distillation approach to compressing the deep Transformer model into a shallow model. The experimental results on several benchmarks validate the effectiveness of our method. Our compressed model is 8X shallower than the deep model, with almost no loss in BLEU. To further enhance the teacher model, we present a Skipping Sub-Layer method to randomly omit sub-layers to introduce perturbation into training, which achieves a BLEU score of 30.63 on English-German newstest2014. The code is publicly available at https://github.com/libeineu/GPKD.
Accepted by AAAI2021
References in corpus (8)
- Distilling the Knowledge in a Neural Network
- Improving neural networks by preventing co-adaptation of feature detectors
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Compressing Deep Convolutional Networks using Vector Quantization
- Pay Less Attention with Lightweight and Dynamic Convolutions
- Highway and Residual Networks learn Unrolled Iterative Estimation
- DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
- Sharing Attention Weights for Fast Transformer