Scalable Transformers for Neural Machine Translation

Abstract

Transformer has been widely adopted in Neural Machine Translation (NMT) because of its large capacity and parallel training of sequence generation. However, the deployment of Transformer is challenging because different scenarios require models of different complexities and scales. Naively training multiple Transformers is redundant in terms of both computation and memory. In this paper, we propose a novel Scalable Transformers, which naturally contains sub-Transformers of different scales and have shared parameters. Each sub-Transformer can be easily obtained by cropping the parameters of the largest Transformer. A three-stage training scheme is proposed to tackle the difficulty of training the Scalable Transformers, which introduces additional supervisions from word-level and sequence-level self-distillation. Extensive experiments were conducted on WMT EN-De and En-Fr to validate our proposed Scalable Transformers.

Mostly overlapping with version 1, with minor updates/revisions

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Language Models are Few-Shot Learners
• Adaptive Computation Time for Recurrent Neural Networks
• Depthwise Separable Convolutions for Neural Machine Translation
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• The Evolved Transformer
• Weighted Transformer Network for Machine Translation
• RomeBERT: Robust Training of Multi-Exit BERT
• Multi-Pass Transformer for Machine Translation