13 citations · 43 across the 10 of their papers we have counts for
11 papers
Survey on Large Scale Neural Network Training
Julia Gusak, Daria Cherniuk, Alena Shilova +8
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU de…
Memory-Efficient Backpropagation through Large Linear Layers
Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa +3
In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to…
Meta-Solver for Neural Ordinary Differential Equations
Julia Gusak, Alexandr Katrutsa, Talgat Daulbaev +2
A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss functio…
Follow the bisector: a simple method for multi-objective optimization
Alexandr Katrutsa, Daniil Merkulov, Nurislam Tursynbek +1
This study presents a novel Equiangular Direction Method (EDM) to solve a multi-objective optimization problem. We consider optimization problems, where multiple differentiable los…
Towards Understanding Normalization in Neural ODEs
Julia Gusak, Larisa Markeeva, Talgat Daulbaev +3
Normalization is an important and vastly investigated technique in deep learning. However, its role for Ordinary Differential Equation based networks (neural ODEs) is still poorly…
Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs
Talgat Daulbaev, Alexandr Katrutsa, Larisa Markeeva +3
We propose a simple interpolation-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method (known in the litera…