4 citations · 5 across the 16 of their papers we have counts for
6 papers · 1 filter
DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
Nikolay Yudin, Ekaterina Grishina, Andrey Veprikov +2
Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured…
Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
Andrey Veprikov, Arman Bolatov, Aleksandr Bogdanov +4
Optimization lies at the core of modern deep learning, yet existing methods often face a fundamental trade-off between adapting to problem geometry and leveraging curvature utiliza…
Aligning Distributionally Robust Optimization with Practical Deep Learning Needs
Dmitrii Feoktistov, Igor Ignashin, Andrey Veprikov +4
While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to di…
Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update
Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov +3
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. However, there is s…
Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
Egor Petrov, Grigoriy Evseev, Aleksey Antonov +4
Fine-tuning Large Language Models (LLMs) is essential for adapting pre-trained models to downstream tasks. Yet traditional first-order optimizers such as Stochastic Gradient Descen…
WeightLoRA: Keep Only Necessary Adapters
Andrey Veprikov, Vladimir Solodkin, Alexander Zyl +2
The widespread utilization of language models in modern applications is inconceivable without Parameter-Efficient Fine-Tuning techniques, such as low-rank adaptation ($\texttt{LoRA…