Very Deep Transformers for Neural Machine Translation
arXiv:2008.07772
Abstract
We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we show that it is feasible to build standard Transformer-based models with up to 60 encoder layers and 12 decoder layers. These deep models outperform their baseline 6-layer counterparts by as much as 2.5 BLEU, and achieve new state-of-the-art benchmark results on WMT14 English-French (43.8 BLEU and 46.4 BLEU with back-translation) and WMT14 English-German (30.1 BLEU).The code and trained models will be publicly available at: https://github.com/namisan/exdeep-nmt.
arXiv comment6 pages, 3 figures and 4 tables. V2 includes the back-translation results
References in corpus (10)
- Language Models are Few-Shot Learners
- Convolutional Sequence to Sequence Learning
- Training Very Deep Networks
- Deep Learning Based Text Classification: A Comprehensive Review
- Pay Less Attention with Lightweight and Dynamic Convolutions
- The Evolved Transformer
- Understanding and Improving Layer Normalization
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- Adversarial Training for Large Neural Language Models
- Understanding the Difficulty of Training Transformers
Cited by in corpus (17)
- R-Drop: Regularized Dropout for Neural Networks
- Grammatical Error Correction: A Survey of the State of the Art
- Accelerating Transformer Inference for Translation via Parallel Decoding
- Understanding the Difficulty of Training Transformers
- Taming Sparsely Activated Transformer with Stochastic Experts
- VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
- On Learning Universal Representations Across Languages
- OmniNet: Omnidirectional Representations from Transformers
- CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
- Controlling Neural Networks with Rule Representations
- A Simple and Effective Positional Encoding for Transformers
- Isomorphic Cross-lingual Embeddings for Low-Resource Languages
- Vocabulary Learning via Optimal Transport for Neural Machine Translation
- WeChat Neural Machine Translation Systems for WMT21
- Cascaded Head-colliding Attention
- Text analysis and deep learning: A network approach
- Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages