8 citations · 10 across the 7 of their papers we have counts for
5 papers · 1 filter
Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression
Viacheslav Yusupov, Daria Cherniuk, Evgeny Frolov
In this paper, we propose a scalable Kronecker-based approximation that captures cross-layer interactions without storing the entire Fisher matrix, enabling practical Hessian analy…
Quantization of Large Language Models with an Overdetermined Basis
Daniil Merkulov, Daria Cherniuk, Alexander Rudikov +4
In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or te…
Run LoRA Run: Faster and Lighter LoRA Implementations
Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets
LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuni…
Quantization Aware Factorization for Deep Neural Network Compression
Daria Cherniuk, Stanislav Abukhovich, Anh-Huy Phan +3
Tensor decomposition of convolutional and fully-connected layers is an effective way to reduce parameters and FLOP in neural networks. Due to memory and power consumption limitatio…
Survey on Large Scale Neural Network Training
Julia Gusak, Daria Cherniuk, Alena Shilova +8
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU de…