Parallel SGD: When does averaging help?

Abstract

Consider a number of workers running SGD independently on the same pool of data and averaging the models every once in a while -- a common but not well understood practice. We study model averaging as a variance-reducing mechanism and describe two ways in which the frequency of averaging affects convergence. For convex objectives, we show the benefit of frequent averaging depends on the gradient variance envelope. For non-convex objectives, we illustrate that this benefit depends on the presence of multiple globally optimal points. We complement our findings with multicore experiments on both synthetic and real data.

Cited by in corpus (44)

• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• On the Convergence of Local Descent Methods in Federated Learning
• A Field Guide to Federated Optimization
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• Communication optimization strategies for distributed deep neural network training: A survey
• Is Local SGD Better than Minibatch SGD?
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
• Faster Neural Network Training with Data Echoing
• Decentralized Deep Learning with Arbitrary Communication Compression
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
• Differentially Private Federated Learning for Resource-Constrained Internet of Things
• Communication trade-offs for synchronized distributed SGD with large step size
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Parallel Restarted SPIDER -- Communication Efficient Distributed Nonconvex Optimization with Optimal Computation Complexity
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Local SGD With a Communication Overhead Depending Only on the Number of Workers
• FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimization
• FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
• A Distributed Hierarchical SGD Algorithm with Sparse Global Reduction
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Communication-efficient SGD: From Local SGD to One-Shot Averaging
• Hierarchical Weight Averaging for Deep Neural Networks
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• On the Convergence of Nested Decentralized Gradient Methods with Multiple Consensus and Gradient Steps
• Distributed Optimization for Over-Parameterized Learning
• Accelerating Gossip SGD with Periodic Global Averaging
• Multi-Level Local SGD for Heterogeneous Hierarchical Networks
• A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to Balance Communication Overhead, Computational Complexity, and Convergence Rate
• Local SGD Optimizes Overparameterized Neural Networks in Polynomial Time
• The Minimax Complexity of Distributed Optimization
• CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Manner
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD
• Weighted parallel SGD for distributed unbalanced-workload training system
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning
• Oscars: Adaptive Semi-Synchronous Parallel Model for Distributed Deep Learning with Global View
• Mage: Online Interference-Aware Scheduling in Multi-Scale Heterogeneous Systems
• Trade-offs of Local SGD at Scale: An Empirical Study
• Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions