10 citations · 78 across the 72 of their papers we have counts for
18 papers · 1 filter
MAST: Model-Agnostic Sparsified Training
Yury Demidovich, Grigory Malinovsky, Egor Shulgin +1
We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function. Unlike traditional f…
Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences
Grigory Malinovsky, Peter Richtárik, Samuel Horváth +1
Distributed learning has emerged as a leading paradigm for training large machine learning models. However, in real-world scenarios, participants may be unreliable or malicious, po…
Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Rates
Ahmad Rammal, Kaja Gruntkowska, Nikita Fedin +2
Byzantine robustness is an essential feature of algorithms for certain distributed optimization problems, typically encountered in collaborative/federated learning. These problems…
Variance Reduced Distributed Non-Convex Optimization Using Matrix Stepsizes
Hanmin Li, Avetik Karagulyan, Peter Richtárik
Matrix-stepsized gradient descent algorithms have been shown to have superior performance in non-convex optimization problems compared to their scalar counterparts. The det-CGD alg…
High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise
Eduard Gorbunov, Abdurakhmon Sadiev, Marina Danilova +5
High-probability analysis of stochastic first-order optimization methods under mild assumptions on the noise has been gaining a lot of attention in recent years. Typically, gradien…
Improving Accelerated Federated Learning with Compression and Importance Sampling
Michał Grudzień, Grigory Malinovsky, Peter Richtárik
Federated Learning is a collaborative training framework that leverages heterogeneous data distributed across a vast number of clients. Since it is practically infeasible to reques…
Towards a Better Theoretical Understanding of Independent Subnetwork Training
Egor Shulgin, Peter Richtárik
Modern advancements in large-scale machine learning would be impossible without the paradigm of data-parallel distributed computing. Since distributed computing with large-scale mo…
Clip21: Error Feedback for Gradient Clipping
Sarit Khirirat, Eduard Gorbunov, Samuel Horváth +3
Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipp…
Error Feedback Shines when Features are Rare
Peter Richtárik, Elnur Gasanov, Konstantin Burlachenko
We provide the first proof that gradient descent with greedy sparsification and error feedback $\left({\…
Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learning
Kai Yi, Laurent Condat, Peter Richtárik
Federated Learning is an evolving machine learning paradigm, in which multiple clients perform computations based on their individual private data, interspersed by communication wi…
Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees
Jihao Xin, Marco Canini, Peter Richtárik +1
Distributed training enables large-scale deep learning, but suffers from high communication overhead, especially as models and datasets grow. Gradient compression, particularly qua…
Momentum Provably Improves Error Feedback!
Ilyas Fatkhullin, Alexander Tyurin, Peter Richtárik
Due to the high communication overhead when training machine learning models in a distributed environment, modern algorithms invariably rely on lossy communication compression. How…
2Direction: Theoretically Faster Distributed Training with Bidirectional Communication Compression
Alexander Tyurin, Peter Richtárik
We consider distributed convex optimization problems in the regime when the communication between the server and the workers is expensive in both uplink and downlink directions. We…
ELF: Federated Langevin Algorithms with Primal, Dual and Bidirectional Compression
Avetik Karagulyan, Peter Richtárik
Federated sampling algorithms have recently gained great popularity in the community of machine learning and statistics. This paper studies variants of such algorithms called Error…
Federated Learning with Regularized Client Participation
Grigory Malinovsky, Samuel Horváth, Konstantin Burlachenko +1
Federated Learning (FL) is a distributed machine learning approach where multiple clients work together to solve a machine learning task. One of the key challenges in FL is the iss…
TAMUNA: Doubly Accelerated Distributed Optimization under Partial Participation
Laurent Condat, Ivan Agarský, Grigory Malinovsky +1
In distributed optimization and federated learning, slow and costly communication between parallel devices and the central server constitutes the primary bottleneck. To alleviate t…
Catalyst Acceleration of Error Compensated Methods Leads to Better Communication Complexity
Xun Qian, Hanze Dong, Tong Zhang +1
Communication overhead is well known to be a key bottleneck in large scale distributed learning, and a particularly successful class of methods which help to overcome this bottlene…
Can 5th Generation Local Training Methods Support Client Sampling? Yes!
Michał Grudzień, Grigory Malinovsky, Peter Richtárik
The celebrated FedAvg algorithm of McMahan et al. (2017) is based on three components: client sampling (CS), data sampling (DS) and local training (LT). While the first two are rea…