Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
arXiv:1808.08859
Abstract
In order to extract the best possible performance from asynchronous stochastic gradient descent one must increase the mini-batch size and scale the learning rate accordingly. In order to achieve further speedup we introduce a technique that delays gradient updates effectively increasing the mini-batch size. Unfortunately with the increase of mini-batch size we worsen the stale gradient problem in asynchronous stochastic gradient descent (SGD) which makes the model convergence poor. We introduce local optimizers which mitigate the stale gradient problem and together with fine tuning our momentum we are able to train a shallow machine translation system 27% faster than an optimized baseline with negligible penalty in BLEU.
To appear in EMNLP 2018 as a short paper
References in corpus (7)
- Neural Machine Translation by Jointly Learning to Align and Translate
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Don't Use Large Mini-Batches, Use Local SGD
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- Deep learning with Elastic Averaging SGD
- Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs
- The University of Edinburgh's Neural MT Systems for WMT17