101 citations · 414 across the 51 of their papers we have counts for
12 papers · 1 filter
On the Role of Batch Size in Stochastic Conditional Gradient Methods
Rustem Islamov, Roman Machacek, Aurelien Lucchi +3
We study the role of batch size in stochastic conditional gradient methods under a -Kurdyka-Łojasiewicz (-KL) condition. Focusing on momentum-based stochastic conditional gra…
On the Interaction of Batch Noise, Adaptivity, and Compression, under -Smoothness: An SDE Approach
Enea Monzio Compagnoni, Rustem Islamov, Frank Norbert Proske +3
Distributed stochastic optimization intertwines (i) stochastic gradient noise, (ii) communication compression, and (iii) adaptive/normalized updates. While each factor has been stu…
Clip21: Error Feedback for Gradient Clipping
Sarit Khirirat, Eduard Gorbunov, Samuel Horváth +3
Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipp…
Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity
Konstantin Mishchenko, Rustem Islamov, Eduard Gorbunov +1
We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training.…
Implicitly normalized forecaster with clipping for linear and non-linear heavy-tailed multi-armed bandits
Yuriy Dorn, Nikita Kornilov, Nikolay Kutuzov +3
The Implicitly Normalized Forecaster (INF) algorithm is considered to be an optimal solution for adversarial multi-armed bandit (MAB) problems. However, most of the existing comple…
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression
Abdurakhmon Sadiev, Grigory Malinovsky, Eduard Gorbunov +4
Gradient compression is a popular technique for improving communication complexity of stochastic first-order methods in distributed training of machine learning models. However, th…
3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation
Peter Richtárik, Igor Sokolov, Ilyas Fatkhullin +3
We propose and study a new class of gradient communication mechanisms for communication-efficient training -- three point compressors (3PC) -- as well as efficient distributed nonc…
Secure Distributed Training at Scale
Eduard Gorbunov, Alexander Borzunov, Michael Diskin +1
Many areas of deep learning benefit from using increasingly larger neural networks trained on public data, as is the case for pre-trained models for NLP and computer vision. Traini…
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk +1
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes. This approach, known as distributed training, can utilize hundreds of comp…
MARINA: Faster Non-Convex Distributed Learning with Compression
Eduard Gorbunov, Konstantin Burlachenko, Zhize Li +1
We develop and analyze MARINA: a new communication efficient method for non-convex distributed learning over heterogeneous datasets. MARINA employs a novel communication compressio…
Local SGD: Unified Theory and New Efficient Methods
Eduard Gorbunov, Filip Hanzely, Peter Richtárik
We present a unified framework for analyzing local SGD methods in the convex and strongly convex regimes for distributed/federated training of supervised machine learning models. W…
Distributed Learning with Compressed Gradient Differences
Konstantin Mishchenko, Eduard Gorbunov, Martin Takáč +1
Training large machine learning models requires a distributed computing approach, with communication of the model updates being the bottleneck. For this reason, several methods bas…