Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning

Abstract

Modern realities and trends in learning require more and more generalization ability of models, which leads to an increase in both models and training sample size. It is already difficult to solve such tasks in a single device mode. This is the reason why distributed and federated learning approaches are becoming more popular every day. Distributed computing involves communication between devices, which requires solving two key problems: efficiency and privacy. One of the most well-known approaches to combat communication costs is to exploit the similarity of local data. Both Hessian similarity and homogeneous gradients have been studied in the literature, but separately. In this paper, we combine both of these assumptions in analyzing a new method that incorporates the ideas of using data similarity and clients sampling. Moreover, to address privacy concerns, we apply the technique of additional noise and analyze its impact on the convergence of the proposed method. The theory is confirmed by training on real datasets.

arXiv comment25 pages, 15 figures, 4 appendices

References in corpus (26)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Support Vector Machines with Applications
• A Universal Catalyst for First-Order Optimization
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient
• Unified Optimal Analysis of the (Stochastic) Gradient Method
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Distributed Robust Learning
• PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization
• A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent
• Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization
• Differential Privacy Made Easy
• Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loop
• Distributed Saddle-Point Problems Under Similarity
• Local SGD: Unified Theory and New Efficient Methods
• On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyond
• Acceleration in Distributed Optimization under Similarity
• A Decentralized Adaptive Momentum Method for Solving a Class of Min-Max Optimization Problems
• Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities
• Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities
• Faster federated optimization under second-order similarity
• Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)
• Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis