Understanding the Difficulty of Training Transformers

Abstract

Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers carefully (e.g., conventional SGD fails to train Transformers effectively). Our objective here is to understand $\text{what complicates Transformer training}$ from both empirical and theoretical perspectives. Our analysis reveals that unbalanced gradients are not the root cause of the instability of training. Instead, we identify an amplification effect that influences training substantially -- for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable, since it amplifies small parameter perturbations (e.g., parameter updates) and results in significant disturbances in the model output. Yet we observe that a light dependency limits the model potential and leads to inferior trained models. Inspired by our analysis, we propose Admin ($\text{Ad}$aptive $\text{m}$odel $\text{in}$itialization) to stabilize stabilize the early stage's training and unleash its full potential in the late stage. Extensive experiments show that Admin is more stable, converges faster, and leads to better performance. Implementations are released at: https://github.com/LiyuanLucasLiu/Transforemr-Clinic.

arXiv commentEMNLP 2020

References in corpus (10)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Layer Normalization
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Training Tips for the Transformer Model
• Image Transformer
• ReZero is All You Need: Fast Convergence at Large Depth
• Very Deep Transformers for Neural Machine Translation
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning

Cited by in corpus (13)

• Zero-Shot Text-to-Image Generation
• Long-Range Transformers for Dynamic Spatiotemporal Forecasting
• Very Deep Transformers for Neural Machine Translation
• NormFormer: Improved Transformer Pretraining with Extra Normalization
• Normalized Attention Without Probability Cage
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
• Group Equivariant Stand-Alone Self-Attention For Vision
• Shallow-to-Deep Training for Neural Machine Translation
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• Self-Supervision is All You Need for Solving Rubik's Cube
• LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction
• Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits