56 citations · 82 across the 8 of their papers we have counts for
9 papers
NNTile: a machine learning framework capable of training extremely large GPT language models on a single node
Aleksandr Mikhalev, Aleksandr Katrutsa, Konstantin Sozykin +1
This study presents an NNTile framework for training large deep neural networks in heterogeneous clusters. The NNTile is based on a StarPU library, which implements task-based para…
Quantization of Large Language Models with an Overdetermined Basis
Daniil Merkulov, Daria Cherniuk, Alexander Rudikov +4
In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or te…
LoTR: Low Tensor Rank Weight Adaptation
Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev +2
In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fin…
Run LoRA Run: Faster and Lighter LoRA Implementations
Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets
LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuni…
Linear scaling computation of forces for the domain-decomposition linear Poisson--Boltzmann method
Abhinav Jha, Michele Nottoli, Aleksandr Mikhalev +2
The Linearized Poisson--Boltzmann (LPB) equation is a popular and widely accepted model for accounting solvent effects in computational (bio-) chemistry. In the present article we…
Memory-Efficient Backpropagation through Large Linear Layers
Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa +3
In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to…
An approximation strategy to compute accurate initial density matrices for repeated self-consistent field calculations at different geometries
E. Polack, A. Mikhalev, Geneviève Dusson +2
Repeated computations on the same molecular system, but with different geometries, are often performed in quantum chemistry, for instance, in ab-initio molecular dynamics simulatio…
Rectangular maximum-volume submatrices and their applications
A. Mikhalev, I. V. Oseledets
We introduce a definition of the volume for a general rectangular matrix, which for square matrices is equivalent to the absolute value of the determinant. We generalize results fo…
Iterative representing set selection for nested cross approximation
A. Yu Mikhalev, I. V. Oseledets
A new fast algebraic method for obtaining an -approximation of a matrix from its entries is presented. The main idea behind the method is based on the nested represe…