10 citations · 78 across the 72 of their papers we have counts for
20 papers · 1 filter
GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity
Artavazd Maranjyan, Mher Safaryan, Peter Richtárik
We study a class of distributed optimization algorithms that aim to alleviate high communication costs by allowing clients to perform multiple local gradient-type training steps be…
CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimization
Laurent Condat, Ivan Agarský, Peter Richtárik
In distributed optimization, a large number of machines alternate between local computations and communication with a coordinating server. Communication, which can be slow and cost…
Personalized Federated Learning with Communication Compression
El Houcine Bergou, Konstantin Burlachenko, Aritra Dutta +1
In contrast to training traditional machine learning (ML) models in data centers, federated learning (FL) trains ML models over local datasets contained on resource-constrained het…
EF21-P and Friends: Improved Theoretical Communication Complexity for Distributed Optimization with Bidirectional Compression
Kaja Gruntkowska, Alexander Tyurin, Peter Richtárik
In this work we focus our attention on distributed optimization problems in the context where the communication time between the server and the workers is non-negligible. We obtain…
Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning
Grigory Malinovsky, Kai Yi, Peter Richtárik
We study distributed optimization methods based on the {\em local training (LT)} paradigm: achieving communication efficiency by performing richer local gradient-based training on…
Shifted Compression Framework: Generalizations and Improvements
Egor Shulgin, Peter Richtárik
Communication is one of the key bottlenecks in the distributed training of large-scale machine learning models, and lossy compression of exchanged information, such as stochastic g…
Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation
Rustem Islamov, Xun Qian, Slavomír Hanzely +2
Despite their high computation and communication costs, Newton-type methods remain an appealing option for distributed training due to their robustness against ill-conditioned conv…
Federated Learning with a Sampling Algorithm under Isoperimetry
Lukang Sun, Adil Salim, Peter Richtárik
Federated learning uses a set of techniques to efficiently distribute the training of a machine learning algorithm across several devices, who own the training data. These techniqu…
Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling
Alexander Tyurin, Lukang Sun, Konstantin Burlachenko +1
We revisit the classical problem of finding an approximately stationary point of the average of smooth and possibly nonconvex functions. The optimal complexity of stochastic fi…
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression
Abdurakhmon Sadiev, Grigory Malinovsky, Eduard Gorbunov +4
Gradient compression is a popular technique for improving communication complexity of stochastic first-order methods in distributed training of machine learning models. However, th…
Certified Robustness in Federated Learning
Motasem Alfarra, Juan C. Pérez, Egor Shulgin +2
Federated learning has recently gained significant attention and popularity due to its effectiveness in training machine learning models on distributed data privately. However, as…
Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top
Eduard Gorbunov, Samuel Horváth, Peter Richtárik +1
Byzantine-robustness has been gaining a lot of attention due to the growth of the interest in collaborative and federated learning. However, many fruitful directions, such as the u…
A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting
Alexander Tyurin, Peter Richtárik
We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, an…
EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization
Laurent Condat, Kai Yi, Peter Richtárik
In distributed or federated optimization and learning, communication between the different computing units is often the bottleneck and gradient compression is widely used to reduce…
FedShuffle: Recipes for Better Use of Local Work in Federated Learning
Samuel Horváth, Maziar Sanjabi, Lin Xiao +2
The practice of applying several local updates before aggregation across clients has been empirically shown to be a successful approach to overcoming the communication bottleneck i…
3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation
Peter Richtárik, Igor Sokolov, Ilyas Fatkhullin +3
We propose and study a new class of gradient communication mechanisms for communication-efficient training -- three point compressors (3PC) -- as well as efficient distributed nonc…
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich +1
We introduce ProxSkip -- a surprisingly simple and provably efficient method for minimizing the sum of a smooth () and an expensive nonsmooth proximable () function. The cano…
DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronization
Alexander Tyurin, Peter Richtárik
We develop and analyze DASHA: a new family of methods for nonconvex distributed optimization problems. When the local functions at the nodes have a finite-sum or an expectation for…
Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimization
Grigory Malinovsky, Konstantin Mishchenko, Peter Richtárik
We present a theoretical study of server-side optimization in federated learning. Our results are the first to show that the widely popular heuristic of scaling the client updates…
BEER: Fast Rate for Decentralized Nonconvex Optimization with Communication Compression
Haoyu Zhao, Boyue Li, Zhize Li +2
Communication efficiency has been widely recognized as the bottleneck for large-scale decentralized machine learning applications in multi-agent or federated environments. To tackl…