Deep Transformers with Latent Depth
arXiv:2009.13102
Abstract
The Transformer model has achieved state-of-the-art performance in many sequence modeling tasks. However, how to leverage model capacity with large or variable depths is still an open challenge. We present a probabilistic framework to automatically learn which layer(s) to use by learning the posterior distributions of layer selection. As an extension of this framework, we propose a novel method to train one shared Transformer network for multilingual machine translation with different layer selection posteriors for each language pair. The proposed method alleviates the vanishing gradient issue and enables stable training of deep Transformers (e.g. 100 layers). We evaluate on WMT English-German machine translation and masked language modeling tasks, where our method outperforms existing approaches for training deeper Transformers. Experiments on multilingual machine translation demonstrate that this approach can effectively leverage increased model capacity and bring universal improvement for both many-to-one and one-to-many translation with diverse language pairs.
References in corpus (17)
- Categorical Reparameterization with Gumbel-Softmax
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- Exponential expressivity in deep neural networks through transient chaos
- Reducing Transformer Depth on Demand with Structured Dropout
- Towards a Human-like Open-Domain Chatbot
- ReZero is All You Need: Fast Convergence at Large Depth
- How multilingual is Multilingual BERT?
- BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
- Fixup Initialization: Residual Learning Without Normalization
- Generalized Inner Loop Meta-Learning
- How to train your MAML
- Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
- Meta Architecture Search
- Training Deeper Neural Machine Translation Models with Transparent Attention
- When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?
- Balancing Training for Multilingual Neural Machine Translation
- Multiscale Collaborative Deep Models for Neural Machine Translation