12 papers
Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
Mikhail Solonko, Molozhavenko Alexander, Maxim Rakhuba
We study Muon, a recently proposed matrix-aware optimization method, in the context of the Stiefel manifold. This manifold consists of matrices with orthonormal columns and is ubiq…
COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Uliana Parkina, Maxim Rakhuba
Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks. In this type of approximati…
Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
Ekaterina Grishina, Matvey Smirnov, Maxim Rakhuba
The problem of computing optimal orthogonal approximation to a given matrix has attracted growing interest in machine learning. Notable applications include the recent Muon optimiz…
Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation
Askar Tsyganov, Evgeny Frolov, Sergey Samsonov +1
In this paper, we propose new randomized algorithms for estimating the two-to-infinity and one-to-two norms in a matrix-free setting, using only matrix-vector multiplications. Our…
DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
Nikolay Yudin, Ekaterina Grishina, Andrey Veprikov +2
Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured…
LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +4
This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank mani…