Making Asynchronous Stochastic Gradient Descent Work for Transformers
arXiv:1906.03496 · doi:10.18653/v1/D19-5608
Abstract
Asynchronous stochastic gradient descent (SGD) is attractive from a speed perspective because workers do not wait for synchronization. However, the Transformer model converges poorly with asynchronous SGD, resulting in substantially lower quality compared to synchronous SGD. To investigate why this is the case, we isolate differences between asynchronous and synchronous methods to investigate batch size and staleness effects. We find that summing several asynchronous updates, rather than applying them immediately, restores convergence behavior. With this hybrid method, Transformer training for neural machine translation task reaches a near-convergence level 1.36x faster in single-node multi-GPU training with no impact on model quality.
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Revisiting Distributed Synchronous SGD
- Don't Decay the Learning Rate, Increase the Batch Size
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- Training Tips for the Transformer Model
- Staleness-aware Async-SGD for Distributed Deep Learning
- Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
- Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation