4 citations · 4 across the 10 of their papers we have counts for
6 papers · 1 filter
Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimization
Yury Demidovich, Abhishek Chakraborty, Grigory Malinovsky +2
Muon and related normalized optimizers decouple the choice of update direction from the choice of step scale, but their practical performance remains sensitive to the scale of the…
Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptation
Igor Sokolov, Abdurakhmon Sadiev, Yury Demidovich +2
Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large foundational models to specific tasks, particularly as model sizes continue to grow expo…
Streamlining in the Riemannian Realm: Efficient Riemannian Optimization with Loopless Variance Reduction
Yury Demidovich, Grigory Malinovsky, Peter Richtárik
In this study, we investigate stochastic optimization on Riemannian manifolds, focusing on the crucial variance reduction mechanism used in both Euclidean and Riemannian settings.…
Correlated Quantization for Faster Nonconvex Distributed Optimization
Andrei Panferov, Yury Demidovich, Ahmad Rammal +1
Quantization (Alistarh et al., 2017) is an important (stochastic) compression technique that reduces the volume of transmitted bits during each communication round in distributed m…
MAST: Model-Agnostic Sparsified Training
Yury Demidovich, Grigory Malinovsky, Egor Shulgin +1
We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function. Unlike traditional f…
A Guide Through the Zoo of Biased SGD
Yury Demidovich, Grigory Malinovsky, Igor Sokolov +1
Stochastic Gradient Descent (SGD) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extens…