8 citations · 20 across the 6 of their papers we have counts for
4 papers · 1 filter
Survey on Large Scale Neural Network Training
Julia Gusak, Daria Cherniuk, Alena Shilova +8
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU de…
Memory-Efficient Backpropagation through Large Linear Layers
Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa +3
In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to…
Meta-Solver for Neural Ordinary Differential Equations
Julia Gusak, Alexandr Katrutsa, Talgat Daulbaev +2
A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss functio…
Towards Understanding Normalization in Neural ODEs
Julia Gusak, Larisa Markeeva, Talgat Daulbaev +3
Normalization is an important and vastly investigated technique in deep learning. However, its role for Ordinary Differential Equation based networks (neural ODEs) is still poorly…