Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Abstract

It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and -- in asynchronous implementations -- on the gradient staleness. Especially, it has been observed that the speedup saturates beyond a certain batch size and/or when the delays grow too large. We identify a data-dependent parameter that explains the speedup saturation in both these settings. Our comprehensive theoretical analysis, for strongly convex, convex and non-convex settings, unifies and generalized prior work directions that often focused on only one of these two aspects. In particular, our approach allows us to derive improved speedup results under frequently considered sparsity assumptions. Our insights give rise to theoretically based guidelines on how the learning rates can be adjusted in practice. We show that our results are tight and illustrate key findings in numerical experiments.

arXiv commentProceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021

References in corpus (13)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• Distributed Delayed Stochastic Optimization
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• Improved asynchronous parallel optimization analysis for stochastic incremental methods
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates
• The Effect of Network Width on the Performance of Large-batch Training
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD

Cited by in corpus (1)

• Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models