14 citations · 16 across the 19 of their papers we have counts for
11 papers · 1 filter
DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
Nikolay Yudin, Ekaterina Grishina, Andrey Veprikov +2
Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured…
Ultra Fast Warm Start Solution for Graph Recommendations
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
In this work, we present a fast and effective Linear approach for updating recommendations in a scalable graph-based recommender system UltraGCN. Solving this task is extremely imp…
Leveraging Geometric Insights in Hyperbolic Triplet Loss for Improved Recommendations
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
Recent studies have demonstrated the potential of hyperbolic geometry for capturing complex patterns from interaction data in recommender systems. In this work, we introduce a nove…
Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation
Askar Tsyganov, Evgeny Frolov, Sergey Samsonov +1
In this paper, we propose new randomized algorithms for estimating the two-to-infinity and one-to-two norms in a matrix-free setting, using only matrix-vector multiplications. Our…
LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +4
This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank mani…
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
Nikolay Yudin, Sergei Kudriashov, Alexander Gaponov +1
We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions. The proposed bou…
COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Uliana Parkina, Maxim Rakhuba
Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks. In this type of approximati…
On the Upper Bounds for the Matrix Spectral Norm
Alexey Naumov, Maxim Rakhuba, Denis Ryapolov +1
We consider the problem of estimating the spectral norm of a matrix using only matrix-vector products. We propose a new Counterbalance estimator that provides upper bounds on the n…
ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations
Ekaterina Grishina, Mikhail Gorbunov, Maxim Rakhuba
Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured…
Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
Ekaterina Grishina, Matvey Smirnov, Maxim Rakhuba
The problem of computing optimal orthogonal approximation to a given matrix has attracted growing interest in machine learning. Notable applications include the recent Muon optimiz…
Knowledge Graph Completion with Mixed Geometry Tensor Factorization
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
In this paper, we propose a new geometric approach for knowledge graph completion via low rank tensor approximation. We augment a pretrained and well-established Euclidean model ba…