22 citations · 50 across the 10 of their papers we have counts for
3 papers · 1 filter
Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch
Xunyi Zhao, Théotime Le Hellard, Lionel Eyraud +2
We propose Rockmate to control the memory requirements when training PyTorch DNN models. Rockmate is an automatic tool that starts from the model code and generates an equivalent m…
Survey on Large Scale Neural Network Training
Julia Gusak, Daria Cherniuk, Alena Shilova +8
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU de…
Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory
Julien Herrmann, Olivier Beaumont, Lionel Eyraud-Dubois +3
This paper introduces a new activation checkpointing method which allows to significantly decrease memory usage when training Deep Neural Networks with the back-propagation algorit…