Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning

Abstract

Recently, local SGD has got much attention and been extensively studied in the distributed learning community to overcome the communication bottleneck problem. However, the superiority of local SGD to minibatch SGD only holds in quite limited situations. In this paper, we study a new local algorithm called Bias-Variance Reduced Local SGD (BVR-L-SGD) for nonconvex distributed optimization. Algorithmically, our proposed bias and variance reduced local gradient estimator fully utilizes small second-order heterogeneity of local objectives and suggests randomly picking up one of the local models instead of taking the average of them when workers are synchronized. Theoretically, under small heterogeneity of local objectives, we show that BVR-L-SGD achieves better communication complexity than both the previous non-local and local methods under mild conditions, and particularly BVR-L-SGD is the first method that breaks the barrier of communication complexity $\Theta (1/\epsilon )$ for general nonconvex smooth objectives when the heterogeneity is small and the local computation budget is large. Numerical results are given to verify the theoretical findings and give empirical evidence of the superiority of our method.

arXiv comment19 pages

References in corpus (17)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning for Mobile Keyboard Prediction
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Optimal Distributed Online Prediction using Mini-Batches
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• On the Convergence of Local Descent Methods in Federated Learning
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Non-convex Finite-Sum Optimization Via SCSG Methods
• AIDE: Fast and Communication Efficient Distributed Optimization
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Parallel Restarted SPIDER -- Communication Efficient Distributed Nonconvex Optimization with Optimal Computation Complexity
• Faster Non-Convex Federated Learning via Global and Local Momentum