56 citations · 82 across the 8 of their papers we have counts for
4 papers · 1 filter
NNTile: a machine learning framework capable of training extremely large GPT language models on a single node
Aleksandr Mikhalev, Aleksandr Katrutsa, Konstantin Sozykin +1
This study presents an NNTile framework for training large deep neural networks in heterogeneous clusters. The NNTile is based on a StarPU library, which implements task-based para…
Quantization of Large Language Models with an Overdetermined Basis
Daniil Merkulov, Daria Cherniuk, Alexander Rudikov +4
In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or te…
Run LoRA Run: Faster and Lighter LoRA Implementations
Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets
LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuni…
Memory-Efficient Backpropagation through Large Linear Layers
Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa +3
In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to…