10 citations · 78 across the 72 of their papers we have counts for
74 papers
SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneity
Liyang Yuan, Yibo Yang, Dandan Guo +2
Federated Learning (FL) is fundamentally challenged by statistical heterogeneity, where non-identically distributed (non-IID) data induces client drift that severely hampers global…
Demystifying Pipeline Parallelism: First Theory for PipeDream
Ivan Ilin, Peter Richtárik
Training modern machine learning models increasingly requires computation to be distributed across many accelerators. Data parallelism remains the default choice and is often paire…
On the Convergence of DP-SGD with Adaptive Clipping
Egor Shulgin, Peter Richtárik
Stochastic Gradient Descent (SGD) with gradient clipping is a powerful technique for enabling differentially private optimization. Although prior works extensively investigated cli…
MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes
Igor Sokolov, Peter Richtárik
Non-smooth communication-efficient federated optimization is crucial for many machine learning applications, yet remains largely unexplored theoretically. Recent advancements have…
Differentially Private Random Block Coordinate Descent
Artavazd Maranjyan, Abdurakhmon Sadiev, Peter Richtárik
Coordinate Descent (CD) methods have gained significant attention in machine learning due to their effectiveness in solving high-dimensional problems and their ability to decompose…
Speeding up Stochastic Proximal Optimization in the High Hessian Dissimilarity Setting
Elnur Gasanov, Peter Richtárik
Stochastic proximal point methods have recently garnered renewed attention within the optimization community, primarily due to their desirable theoretical properties. Notably, thes…
Methods with Local Steps and Random Reshuffling for Generally Smooth Non-Convex Federated Optimization
Yury Demidovich, Petr Ostroukhov, Grigory Malinovsky +4
Non-convex Machine Learning problems typically do not adhere to the standard smoothness assumption. Based on empirical findings, Zhang et al. (2020b) proposed a more realistic gene…
Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
Vladimir Malinovskii, Andrei Panferov, Ivan Ilin +3
Quantizing large language models has become a standard way to reduce their memory and computational costs. Typically, existing methods focus on breaking down the problem into indiv…
Error Feedback under -Smoothness: Normalization and Momentum
Sarit Khirirat, Abdurakhmon Sadiev, Artem Riabinin +2
We provide the first proof of convergence for normalized error feedback algorithms across a wide range of machine learning problems. Despite their popularity and efficiency in trai…
Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation
Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud +4
Fine-tuning has become a popular approach to adapting large foundational models to specific tasks. As the size of models and datasets grows, parameter-efficient fine-tuning techniq…
On the Convergence of FedProx with Extrapolation and Inexact Prox
Hanmin Li, Peter Richtárik
Enhancing the FedProx federated learning algorithm (Li et al., 2020) with server-side extrapolation, Li et al. (2024a) recently introduced the FedExProx method. Their theoretical a…
Tighter Performance Theory of FedExProx
Wojciech Anyszka, Kaja Gruntkowska, Alexander Tyurin +1
We revisit FedExProx - a recently proposed distributed optimization method designed to enhance convergence properties of parallel proximal algorithms via extrapolation. In the proc…
Unlocking FedNL: Self-Contained Compute-Optimized Implementation
Konstantin Burlachenko, Peter Richtárik
Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner, eliminating the need…
MindFlayer SGD: Efficient Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times
Artavazd Maranjyan, Omar Shaikh Omar, Peter Richtárik
We investigate the problem of minimizing the expectation of smooth nonconvex functions in a distributed setting with multiple parallel workers that are able to compute stochastic g…
Methods for Convex -Smooth Optimization: Clipping, Acceleration, and Adaptivity
Eduard Gorbunov, Nazarii Tupitsa, Sayantan Choudhury +4
Due to the non-smoothness of optimization problems in Machine Learning, generalized smoothness assumptions have been gaining a lot of attention in recent years. One of the most pop…
Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning
Kai Yi, Timur Kharisov, Igor Sokolov +1
Virtually all federated learning (FL) methods, including FedAvg, operate in the following manner: i) an orchestrating server sends the current model parameters to a cohort of clien…
SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning
Avetik Karagulyan, Egor Shulgin, Abdurakhmon Sadiev +1
Cross-device training is a crucial subfield of federated learning, where the number of clients can reach into the billions. Standard approaches and local methods are prone to issue…
A Simple Linear Convergence Analysis of the Point-SAGA Algorithm
Laurent Condat, Peter Richtárik
Point-SAGA is a randomized algorithm for minimizing a sum of convex functions using their proximity operators (proxs), proposed by Defazio (2016). At every iteration, the prox of o…
Local Curvature Descent: Squeezing More Curvature out of Standard and Polyak Gradient Descent
Peter Richtárik, Simone Maria Giancola, Dymitr Lubczyk +1
We contribute to the growing body of knowledge on more powerful and adaptive stepsizes for convex optimization, empowered by local curvature information. We do not go the route of…
A Unified Theory of Stochastic Proximal Point Methods without Smoothness
Peter Richtárik, Abdurakhmon Sadiev, Yury Demidovich
This paper presents a comprehensive analysis of a broad range of variations of the stochastic proximal point method (SPPM). Proximal point methods have attracted considerable inter…
Stochastic Proximal Point Methods for Monotone Inclusions under Expected Similarity
Abdurakhmon Sadiev, Laurent Condat, Peter Richtárik
Monotone inclusions have a wide range of applications, including minimization, saddle-point, and equilibria problems. We introduce new stochastic algorithms, with or without varian…
Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning
Georg Meinhardt, Kai Yi, Laurent Condat +1
In Federated Learning (FL), both client resource constraints and communication costs pose major problems for training large models. In the centralized setting, sparse training addr…
On the Optimal Time Complexities in Decentralized Stochastic Asynchronous Optimization
Alexander Tyurin, Peter Richtárik
We consider the decentralized stochastic asynchronous optimization setup, where many workers asynchronously calculate stochastic gradients and asynchronously communicate with each…
Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computations
Alexander Tyurin, Kaja Gruntkowska, Peter Richtárik
In practical distributed systems, workers are typically not homogeneous, and due to differences in hardware configurations and network conditions, can have highly varying processin…