40 citations · 53 across the 10 of their papers we have counts for
10 papers
SILAGE: Memory-Efficient, Full-Gradient-Free Nonconvex Optimization for Nested Finite Sums
Igor Sokolov, Laurent Condat, Peter Richtárik
Empirical risk minimization on massive datasets naturally exhibits a nested double finite-sum structure, where total samples are logically or physically partitioned into …
Improved Convergence in Parameter-Agnostic Error Feedback through Momentum
Abdurakhmon Sadiev, Yury Demidovich, Igor Sokolov +3
Communication compression is essential for scalable distributed training of modern machine learning models, but it often degrades convergence due to the noise it introduces. Error…
Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptation
Igor Sokolov, Abdurakhmon Sadiev, Yury Demidovich +2
Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large foundational models to specific tasks, particularly as model sizes continue to grow expo…
MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes
Igor Sokolov, Peter Richtárik
Non-smooth communication-efficient federated optimization is crucial for many machine learning applications, yet remains largely unexplored theoretically. Recent advancements have…
Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning
Kai Yi, Timur Kharisov, Igor Sokolov +1
Virtually all federated learning (FL) methods, including FedAvg, operate in the following manner: i) an orchestrating server sends the current model parameters to a cohort of clien…
A Guide Through the Zoo of Biased SGD
Yury Demidovich, Grigory Malinovsky, Igor Sokolov +1
Stochastic Gradient Descent (SGD) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extens…
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression
Abdurakhmon Sadiev, Grigory Malinovsky, Eduard Gorbunov +4
Gradient compression is a popular technique for improving communication complexity of stochastic first-order methods in distributed training of machine learning models. However, th…
3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation
Peter Richtárik, Igor Sokolov, Ilyas Fatkhullin +3
We propose and study a new class of gradient communication mechanisms for communication-efficient training -- three point compressors (3PC) -- as well as efficient distributed nonc…
EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback
Ilyas Fatkhullin, Igor Sokolov, Eduard Gorbunov +2
First proposed by Seide (2014) as a heuristic, error feedback (EF) is a very popular mechanism for enforcing convergence of distributed gradient-based optimization methods enhanced…
EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
Peter Richtárik, Igor Sokolov, Ilyas Fatkhullin
Error feedback (EF), also known as error compensation, is an immensely popular convergence stabilization mechanism in the context of distributed training of supervised machine lear…