Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning

Abstract

Modern realities and trends in learning require more and more generalization ability of models, which leads to an increase in both models and training sample size. It is already difficult to solve such tasks in a single device mode. This is the reason why distributed and federated learning approaches are becoming more popular every day. Distributed computing involves communication between devices, which requires solving two key problems: efficiency and privacy. One of the most well-known approaches to combat communication costs is to exploit the similarity of local data. Both Hessian similarity and homogeneous gradients have been studied in the literature, but separately. In this paper, we combine both of these assumptions in analyzing a new method that incorporates the ideas of using data similarity and clients sampling. Moreover, to address privacy concerns, we apply the technique of additional noise and analyze its impact on the convergence of the proposed method. The theory is confirmed by training on real datasets.

25 pages, 15 figures, 4 appendices

References in corpus (16)

• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Distributed Robust Learning
• PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization
• Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization
• Differential Privacy Made Easy
• Distributed Saddle-Point Problems Under Similarity
• Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loop
• Local SGD: Unified Theory and New Efficient Methods
• Acceleration in Distributed Optimization under Similarity
• A Decentralized Adaptive Momentum Method for Solving a Class of Min-Max Optimization Problems
• Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities
• Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities
• Faster federated optimization under second-order similarity
• Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)
• Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis