Local Methods with Adaptivity via Scaling
arXiv:2406.00846 · doi:10.4213/rm10209
Abstract
The rapid development of machine learning and deep learning has introduced increasingly complex optimization challenges that must be addressed. Indeed, training modern, advanced models has become difficult to implement without leveraging multiple computing nodes in a distributed environment. Distributed optimization is also fundamental to emerging fields such as federated learning. Specifically, there is a need to organize the training process to minimize the time lost due to communication. A widely used and extensively researched technique to mitigate the communication bottleneck involves performing local training before communication. This approach is the focus of our paper. Concurrently, adaptive methods that incorporate scaling, notably led by Adam, have gained significant popularity in recent years. Therefore, this paper aims to merge the local training technique with the adaptive approach to develop efficient distributed learning methods. We consider the classical Local SGD method and enhance it with a scaling feature. A crucial aspect is that the scaling is described generically, allowing us to analyze various approaches, including Adam, RMSProp, and OASIS, in a unified manner. In addition to theoretical analysis, we validate the performance of our methods in practice by training a neural network.
41 pages, 2 algorithms, 6 figures, 1 table
References in corpus (25)
- Adam: A Method for Stochastic Optimization
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
- FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
- A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
- Distributed Learning with Compressed Gradient Differences
- SGD and Hogwild! Convergence Without the Bounded Gradients Assumption
- Variance Reduced Local SGD with Lower Communication Complexity
- CoCoA: A General Framework for Communication-Efficient Distributed Optimization
- SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
- Tighter Theory for Local SGD on Identical and Heterogeneous Data
- A Simple Convergence Proof of Adam and Adagrad
- Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization
- Parallel Restarted SPIDER -- Communication Efficient Distributed Nonconvex Optimization with Optimal Computation Complexity
- Distributed Saddle-Point Problems Under Similarity
- ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
- Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
- Distributed Fixed Point Methods with Compressed Iterates
- Optimal Gradient Sliding and its Application to Distributed Optimization Under Similarity
- Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information
- On Scaled Methods for Saddle Point Problems
- Stochastic Gradient Methods with Preconditioned Updates