5 papers
OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models
Ali Aliev, Kamil Garifullin, Nikolay Yudin +5
In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training da…
Mitigating Position-Shift Failures in Text-Based Modular Arithmetic via Position Curriculum and Template Diversity
Nikolay Yudin
Building on insights from the grokking literature, we study character-level Transformers trained to compute modular addition from text, and focus on robustness under input-format v…
DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
Nikolay Yudin, Ekaterina Grishina, Andrey Veprikov +2
Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured…
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
Nikolay Yudin, Sergei Kudriashov, Alexander Gaponov +1
We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions. The proposed bou…
Group and Shuffle: Efficient Structured Orthogonal Parametrization
Mikhail Gorbunov, Nikolay Yudin, Vera Soboleva +3
The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses ortho…