5 papers
Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates
Roman Maksimov, Vladimir Aletov, Dmitry Bylinkin +3
Knowledge editing (KE) provides a lightweight alternative to repeated fine-tuning of LLMs. However, most existing KE methods target dense feed-forward layers, while modern LLMs inc…
Methods for Solving Variational Inequalities with Markovian Stochasticity
Vladimir Solodkin, Michael Ermoshin, Roman Gavrilenko +1
In this paper, we present a novel stochastic method for solving variational inequalities (VI) in the context of Markovian noise. By leveraging Extragradient technique, we can produ…
Markovian Compression: Looking to the Past Helps Accelerate the Future
Andrey Veprikov, Vladimir Solodkin, Mikhail Rudakov +2
This paper deals with distributed optimization problems that use compressed communication to achieve efficient performance and mitigate communication bottleneck. We propose a famil…
Methods for Optimization Problems with Markovian Stochasticity and Non-Euclidean Geometry
Vladimir Solodkin, Andrew Veprikov, Aleksandr Beznosikov
This paper examines a variety of classical optimization problems, including well-known minimization tasks and more general variational inequalities. We consider a stochastic formul…
WeightLoRA: Keep Only Necessary Adapters
Andrey Veprikov, Vladimir Solodkin, Alexander Zyl +2
The widespread utilization of language models in modern applications is inconceivable without Parameter-Efficient Fine-Tuning techniques, such as low-rank adaptation ($\texttt{LoRA…