10 citations · 39 across the 11 of their papers we have counts for
10 papers · 1 filter
Quantization Aware Factorization for Deep Neural Network Compression
Daria Cherniuk, Stanislav Abukhovich, Anh-Huy Phan +3
Tensor decomposition of convolutional and fully-connected layers is an effective way to reduce parameters and FLOP in neural networks. Due to memory and power consumption limitatio…
Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch
Xunyi Zhao, Théotime Le Hellard, Lionel Eyraud +2
We propose Rockmate to control the memory requirements when training PyTorch DNN models. Rockmate is an automatic tool that starts from the model code and generates an equivalent m…
Survey on Large Scale Neural Network Training
Julia Gusak, Daria Cherniuk, Alena Shilova +8
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU de…
Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction
Georgii Novikov, Daniel Bershatsky, Julia Gusak +3
Memory footprint is one of the main limiting factors for large neural network training. In backpropagation, one needs to store the input to each operation in the computational grap…
Memory-Efficient Backpropagation through Large Linear Layers
Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa +3
In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to…
Meta-Solver for Neural Ordinary Differential Equations
Julia Gusak, Alexandr Katrutsa, Talgat Daulbaev +2
A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss functio…
Towards Understanding Normalization in Neural ODEs
Julia Gusak, Larisa Markeeva, Talgat Daulbaev +3
Normalization is an important and vastly investigated technique in deep learning. However, its role for Ordinary Differential Equation based networks (neural ODEs) is still poorly…
Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
Chunfeng Cui, Kaiqi Zhang, Talgat Daulbaev +3
Active subspace is a model reduction method widely used in the uncertainty quantification community. In this paper, we propose analyzing the internal structure and vulnerability an…
Reduced-Order Modeling of Deep Neural Networks
Julia Gusak, Talgat Daulbaev, Evgeny Ponomarev +2
We introduce a new method for speeding up the inference of deep neural networks. It is somewhat inspired by the reduced-order modeling techniques for dynamical systems.The cornerst…
MUSCO: Multi-Stage Compression of neural networks
Julia Gusak, Maksym Kholiavchenko, Evgeny Ponomarev +3
The low-rank tensor approximation is very promising for the compression of deep neural networks. We propose a new simple and efficient iterative approach, which alternates low-rank…