14 citations · 16 across the 19 of their papers we have counts for
12 papers · 1 filter
Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Ali Aliev, Maxim Rakhuba
In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recen…
DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
Nikolay Yudin, Ekaterina Grishina, Andrey Veprikov +2
Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured…
Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation
Askar Tsyganov, Evgeny Frolov, Sergey Samsonov +1
In this paper, we propose new randomized algorithms for estimating the two-to-infinity and one-to-two norms in a matrix-free setting, using only matrix-vector multiplications. Our…
LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +4
This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank mani…
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
Nikolay Yudin, Sergei Kudriashov, Alexander Gaponov +1
We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions. The proposed bou…
COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Uliana Parkina, Maxim Rakhuba
Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks. In this type of approximati…
Knowledge Graph Completion with Mixed Geometry Tensor Factorization
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
In this paper, we propose a new geometric approach for knowledge graph completion via low rank tensor approximation. We augment a pretrained and well-established Euclidean model ba…
Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers
Ekaterina Grishina, Mikhail Gorbunov, Maxim Rakhuba
Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in…
Group and Shuffle: Efficient Structured Orthogonal Parametrization
Mikhail Gorbunov, Nikolay Yudin, Vera Soboleva +3
The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses ortho…
Towards Practical Control of Singular Values of Convolutional Layers
Alexandra Senderovich, Ekaterina Bulatova, Anton Obukhov +1
In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Re…
Spectral Tensor Train Parameterization of Deep Learning Layers
Anton Obukhov, Maxim Rakhuba, Alexander Liniger +4
We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permi…
T-Basis: a Compact Representation for Neural Networks
Anton Obukhov, Maxim Rakhuba, Stamatios Georgoulis +3
We introduce T-Basis, a novel concept for a compact representation of a set of tensors, each of an arbitrary shape, which is often seen in Neural Networks. Each of the tensors in t…