4 citations · 4 across the 10 of their papers we have counts for
10 papers
Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimization
Yury Demidovich, Abhishek Chakraborty, Grigory Malinovsky +2
Muon and related normalized optimizers decouple the choice of update direction from the choice of step scale, but their practical performance remains sensitive to the scale of the…
Better LMO-based Momentum Methods with Second-Order Information
Sarit Khirirat, Abdurakhmon Sadiev, Yury Demidovich +1
The use of momentum in stochastic optimization algorithms has shown empirical success across a range of machine learning tasks. Recently, a new class of stochastic momentum algorit…
Improved Convergence in Parameter-Agnostic Error Feedback through Momentum
Abdurakhmon Sadiev, Yury Demidovich, Igor Sokolov +3
Communication compression is essential for scalable distributed training of modern machine learning models, but it often degrades convergence due to the noise it introduces. Error…
Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptation
Igor Sokolov, Abdurakhmon Sadiev, Yury Demidovich +2
Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large foundational models to specific tasks, particularly as model sizes continue to grow expo…
Methods with Local Steps and Random Reshuffling for Generally Smooth Non-Convex Federated Optimization
Yury Demidovich, Petr Ostroukhov, Grigory Malinovsky +4
Non-convex Machine Learning problems typically do not adhere to the standard smoothness assumption. Based on empirical findings, Zhang et al. (2020b) proposed a more realistic gene…
A Unified Theory of Stochastic Proximal Point Methods without Smoothness
Peter Richtárik, Abdurakhmon Sadiev, Yury Demidovich
This paper presents a comprehensive analysis of a broad range of variations of the stochastic proximal point method (SPPM). Proximal point methods have attracted considerable inter…
Streamlining in the Riemannian Realm: Efficient Riemannian Optimization with Loopless Variance Reduction
Yury Demidovich, Grigory Malinovsky, Peter Richtárik
In this study, we investigate stochastic optimization on Riemannian manifolds, focusing on the crucial variance reduction mechanism used in both Euclidean and Riemannian settings.…
Correlated Quantization for Faster Nonconvex Distributed Optimization
Andrei Panferov, Yury Demidovich, Ahmad Rammal +1
Quantization (Alistarh et al., 2017) is an important (stochastic) compression technique that reduces the volume of transmitted bits during each communication round in distributed m…
MAST: Model-Agnostic Sparsified Training
Yury Demidovich, Grigory Malinovsky, Egor Shulgin +1
We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function. Unlike traditional f…
A Guide Through the Zoo of Biased SGD
Yury Demidovich, Grigory Malinovsky, Igor Sokolov +1
Stochastic Gradient Descent (SGD) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extens…