8 citations · 15 across the 6 of their papers we have counts for
7 papers
On the Spatial Structure of Mixture-of-Experts in Transformers
Daniel Bershatsky, Ivan Oseledets
A common assumption is that MoE routers primarily leverage semantic features for expert selection. However, our study challenges this notion by demonstrating that positional token…
LoTR: Low Tensor Rank Weight Adaptation
Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev +2
In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fin…
Federated Privacy-preserving Collaborative Filtering for On-Device Next App Prediction
Albert Sayapin, Gleb Balitskiy, Daniel Bershatsky +5
In this study, we propose a novel SeqMF model to solve the problem of predicting the next app launch during mobile device usage. Although this problem can be represented as a class…
NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer
Valentin Leplat, Daniil Merkulov, Aleksandr Katrutsa +3
Classical machine learning models such as deep neural networks are usually trained by using Stochastic Gradient Descent-based (SGD) algorithms. The classical SGD can be interpreted…
Survey on Large Scale Neural Network Training
Julia Gusak, Daria Cherniuk, Alena Shilova +8
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU de…
Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction
Georgii Novikov, Daniel Bershatsky, Julia Gusak +3
Memory footprint is one of the main limiting factors for large neural network training. In backpropagation, one needs to store the input to each operation in the computational grap…
Memory-Efficient Backpropagation through Large Linear Layers
Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa +3
In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to…