Trade-offs of Local SGD at Scale: An Empirical Study

Abstract

As datasets and models become increasingly large, distributed training has become a necessary component to allow deep neural networks to train in reasonable amounts of time. However, distributed training can have substantial communication overhead that hinders its scalability. One strategy for reducing this overhead is to perform multiple unsynchronized SGD steps independently on each worker between synchronization steps, a technique known as local SGD. We conduct a comprehensive empirical study of local SGD and related methods on a large-scale image classification task. We find that performing local SGD comes at a price: lower communication costs (and thereby faster training) are accompanied by lower accuracy. This finding is in contrast from the smaller-scale experiments in prior work, suggesting that local SGD encounters challenges at scale. We further show that incorporating the slow momentum framework of Wang et al. (2020) consistently improves accuracy without requiring additional communication, hinting at future directions for potentially escaping this trade-off.

References in corpus (20)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Optimal Distributed Online Prediction using Mini-Batches
• Large Batch Training of Convolutional Networks
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Measuring the Effects of Data Parallelism on Neural Network Training
• Stochastic Gradient Push for Distributed Deep Learning
• Parallel training of DNNs with Natural Gradient and Parameter Averaging
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• Parallel SGD: When does averaging help?
• Experiments on Parallel Training of Deep Neural Network using Model Averaging
• Is Local SGD Better than Minibatch SGD?
• Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
• Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
• Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimization
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs