Taming Momentum in a Distributed Asynchronous Environment

Abstract

Although distributed computing can significantly reduce the training time of deep neural networks, scaling the training process while maintaining high efficiency and final accuracy is challenging. Distributed asynchronous training enjoys near-linear speedup, but asynchrony causes gradient staleness - the main difficulty in scaling stochastic gradient descent to large clusters. Momentum, which is often used to accelerate convergence and escape local minima, exacerbates the gradient staleness, thereby hindering convergence. We propose DANA: a novel technique for asynchronous distributed SGD with momentum that mitigates gradient staleness by computing the gradient on an estimated future position of the model's parameters. Thereby, we show for the first time that momentum can be fully incorporated in asynchronous training with almost no ramifications to final accuracy. Our evaluation on the CIFAR and ImageNet datasets shows that DANA outperforms existing methods, in both final accuracy and convergence speed while scaling up to a total batch size of 16K on 64 asynchronous workers.

References in corpus (28)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Revisiting Distributed Synchronous SGD
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• PyTorch-BigGraph: A Large-scale Graph Embedding System
• Staleness-aware Async-SGD for Distributed Deep Learning
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Beyond Data and Model Parallelism for Deep Neural Networks
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• signSGD: Compressed Optimisation for Non-Convex Problems
• Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• YellowFin and the Art of Momentum Tuning
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• 3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning
• XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
• Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
• Linearly convergent stochastic heavy ball method for minimizing generalization error
• Blink: Fast and Generic Collectives for Distributed ML
• Toward Understanding the Impact of Staleness in Distributed Machine Learning
• Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers
• Pipelined Backpropagation at Scale: Training Large Models without Batches

Cited by in corpus (5)

• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
• ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training
• Gap Aware Mitigation of Gradient Staleness