14 citations · 16 across the 19 of their papers we have counts for
22 papers
Reasoning on the Simplex: Geometric Fixed-Point Models
Talgat Daulbaev, Ilya Glazkov, Maxim Rakhuba +1
Looped reasoners spend test-time compute by iterating a weight-tied map, but a small residual does not mean the state is a fixed point when that map lives in unconstrained latent s…
Fast Differentiable SVD on GPU via Polar Decomposition
Uliana Parkina, Askar Tsyganov, Sergei Kudriashov +2
We present a fully GPU-oriented SVD pipeline based on polar decomposition, motivated by iterative methods that rely solely on matrix multiplications, such as the Newton-Schulz iter…
Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Ali Aliev, Maxim Rakhuba
In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recen…
A lower bound for matrix multiplication
Askar Tsyganov, Uliana Parkina, Sergey Samsonov +1
We prove that, over any field, the bilinear complexity of multiplying a matrix by a matrix is strictly greater than . In particular, every exact bili…
Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
Mikhail Solonko, Molozhavenko Alexander, Maxim Rakhuba
We study Muon, a recently proposed matrix-aware optimization method, in the context of the Stiefel manifold. This manifold consists of matrices with orthonormal columns and is ubiq…
DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
Nikolay Yudin, Ekaterina Grishina, Andrey Veprikov +2
Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured…
Ultra Fast Warm Start Solution for Graph Recommendations
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
In this work, we present a fast and effective Linear approach for updating recommendations in a scalable graph-based recommender system UltraGCN. Solving this task is extremely imp…
Leveraging Geometric Insights in Hyperbolic Triplet Loss for Improved Recommendations
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
Recent studies have demonstrated the potential of hyperbolic geometry for capturing complex patterns from interaction data in recommender systems. In this work, we introduce a nove…
Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation
Askar Tsyganov, Evgeny Frolov, Sergey Samsonov +1
In this paper, we propose new randomized algorithms for estimating the two-to-infinity and one-to-two norms in a matrix-free setting, using only matrix-vector multiplications. Our…
LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +4
This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank mani…
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
Nikolay Yudin, Sergei Kudriashov, Alexander Gaponov +1
We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions. The proposed bou…
COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Uliana Parkina, Maxim Rakhuba
Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks. In this type of approximati…
On the Upper Bounds for the Matrix Spectral Norm
Alexey Naumov, Maxim Rakhuba, Denis Ryapolov +1
We consider the problem of estimating the spectral norm of a matrix using only matrix-vector products. We propose a new Counterbalance estimator that provides upper bounds on the n…
ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations
Ekaterina Grishina, Mikhail Gorbunov, Maxim Rakhuba
Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured…
Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
Ekaterina Grishina, Matvey Smirnov, Maxim Rakhuba
The problem of computing optimal orthogonal approximation to a given matrix has attracted growing interest in machine learning. Notable applications include the recent Muon optimiz…
Knowledge Graph Completion with Mixed Geometry Tensor Factorization
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
In this paper, we propose a new geometric approach for knowledge graph completion via low rank tensor approximation. We augment a pretrained and well-established Euclidean model ba…
Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers
Ekaterina Grishina, Mikhail Gorbunov, Maxim Rakhuba
Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in…
Group and Shuffle: Efficient Structured Orthogonal Parametrization
Mikhail Gorbunov, Nikolay Yudin, Vera Soboleva +3
The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses ortho…
Dimension-free Structured Covariance Estimation
Nikita Puchkin, Maxim Rakhuba
Given a sample of i.i.d. high-dimensional centered random vectors, we consider a problem of estimation of their covariance matrix with an additional assumption that can be…
Towards Practical Control of Singular Values of Convolutional Layers
Alexandra Senderovich, Ekaterina Bulatova, Anton Obukhov +1
In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Re…
Spectral Tensor Train Parameterization of Deep Learning Layers
Anton Obukhov, Maxim Rakhuba, Alexander Liniger +4
We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permi…
T-Basis: a Compact Representation for Neural Networks
Anton Obukhov, Maxim Rakhuba, Stamatios Georgoulis +3
We introduce T-Basis, a novel concept for a compact representation of a set of tensors, each of an arbitrary shape, which is often seen in Neural Networks. Each of the tensors in t…