47 citations · 257 across the 74 of their papers we have counts for
77 papers
The Kuramoto Neural Operator: Learning to Solve PDEs via Coupled Oscillator Dynamics
Petr Badolia, Leonid Obukhov, Dmitry Bylinkin +1
Operator learning is a rapidly advancing area of computational science. It is particularly well suited to problems where a partial differential equation (PDE) must be solved repeat…
Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov +3
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we ad…
Exploring New Frontiers in Vertical Federated Learning: the Role of Saddle Point Reformulation
Aleksandr Beznosikov, Georgiy Kormakov, Alexander Grigorievskiy +7
The objective of Vertical Federated Learning (VFL) is to collectively train a model using features available on different devices while sharing the same users. This paper focuses o…
Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers
Artem Riabinin, Andrey Veprikov, Arman Bolatov +2
We study adaptive learning rate scheduling for norm-constrained optimizers (e.g., Muon and Lion). We introduce a generalized smoothness assumption under which local curvature decre…
Accelerated Methods with Complexity Separation Under Data Similarity for Federated Learning Problems
Dmitry Bylinkin, Sergey Skorik, Dmitriy Bystrov +3
Heterogeneity within data distribution poses a challenge in many modern federated learning tasks. We formalize it as an optimization problem involving a computationally heavy compo…
Markovian Compression: Looking to the Past Helps Accelerate the Future
Andrey Veprikov, Vladimir Solodkin, Mikhail Rudakov +2
This paper deals with distributed optimization problems that use compressed communication to achieve efficient performance and mitigate communication bottleneck. We propose a famil…
Gradient-Free Approaches is a Key to an Efficient Interaction with Markovian Stochasticity
Boris Prokhorov, Semyon Chebykin, Alexander Gasnikov +1
This paper deals with stochastic optimization problems involving Markovian noise with a zero-order oracle. We present and analyze a novel derivative-free method for solving such pr…
Adaptive Regularized Newton Method with Inexact Hessian
Aleksandr Shestakov, Nail Bashirov, Andrei Semenov +4
Newton's method is the most widespread high-order method, demanding the gradient and the Hessian of the objective function. However, one of the main disadvantages of Newtons method…
Unified Theory of Adaptive Variance Reduction
Aleksandr Shestakov, Valery Parfenov, Aleksandr Beznosikov
Variance reduction is a family of powerful mechanisms for stochastic optimization that appears to be helpful in many machine learning tasks. It is based on estimating the exact gra…
Shuffling Heuristic in Variational Inequalities: Establishing New Convergence Guarantees
Daniil Medyakov, Gleb Molodtsov, Grigoriy Evseev +2
Variational inequalities have gained significant attention in machine learning and optimization research. While stochastic methods for solving these problems typically assume indep…
Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
Andrey Veprikov, Arman Bolatov, Aleksandr Bogdanov +4
Optimization lies at the core of modern deep learning, yet existing methods often face a fundamental trade-off between adapting to problem geometry and leveraging curvature utiliza…
Aligning Distributionally Robust Optimization with Practical Deep Learning Needs
Dmitrii Feoktistov, Igor Ignashin, Andrey Veprikov +4
While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to di…
Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update
Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov +3
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. However, there is s…
Communication-Efficient Federated Learning with Adaptive Number of Participants
Sergey Skorik, Vladislav Dorofeev, Gleb Molodtsov +4
Rapid scaling of deep learning models has enabled performance gains across domains, yet it introduced several challenges. Federated Learning (FL) has emerged as a promising framewo…
Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation
Dmitry Bylinkin, Mikhail Aleksandrov, Savelii Chezhegov +1
Physics-informed neural networks (PINNs) have gained prominence in recent years and are now effectively used in a number of applications. However, their performance remains unstabl…
Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
Egor Petrov, Grigoriy Evseev, Aleksey Antonov +4
Fine-tuning Large Language Models (LLMs) is essential for adapting pre-trained models to downstream tasks. Yet traditional first-order optimizers such as Stochastic Gradient Descen…
Sign-SGD via Parameter-Free Optimization
Daniil Medyakov, Sergey Stanko, Gleb Molodtsov +4
Large language models have achieved major advances across domains, yet training them remains extremely resource-intensive. We revisit Sign-SGD, which serves both as a memory-effici…
WeightLoRA: Keep Only Necessary Adapters
Andrey Veprikov, Vladimir Solodkin, Alexander Zyl +2
The widespread utilization of language models in modern applications is inconceivable without Parameter-Efficient Fine-Tuning techniques, such as low-rank adaptation ($\texttt{LoRA…
Incorporating Preconditioning into Accelerated Approaches: Theoretical Guarantees and Practical Improvement
Stepan Trifonov, Leonid Levin, Savelii Chezhegov +1
Machine learning and deep learning are widely researched fields that provide solutions to many modern problems. Due to the complexity of new problems related to the size of dataset…
High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and -Smoothness
Savelii Chezhegov, Taha El Bakkali El Kadi, Aleksandr Beznosikov +2
Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees over stochastic gradient descent (SGD) und…
Bant: Byzantine Antidote via Trial Function and Trust Scores
Gleb Molodtsov, Daniil Medyakov, Sergey Skorik +6
Recent advancements in machine learning have improved performance while also increasing computational demands. While federated and distributed setups address these issues, their st…
Thinking like a CHEMIST: Combined Heterogeneous Embedding Model Integrating Structure and Tokens
Nikolai Rekut, Alexey Orlov, Klea Ziu +3
Representing molecular structures effectively in chemistry remains a challenging task. Language models and graph-based models are extensively utilized within this domain, consisten…
Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shuffling
Daniil Medyakov, Gleb Molodtsov, Savelii Chezhegov +2
Stochastic optimization algorithms are widely used for machine learning with large-scale data. However, their convergence often suffers from non-vanishing variance. Variance Reduct…
Label Privacy in Split Learning for Large Models with Parameter-Efficient Training
Philip Zmushko, Marat Mansurov, Ruslan Svirschevski +3
As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. These web services allow fine-tuning a model between two parties: the client…