Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory

Julien Herrmann, Olivier Beaumont, Lionel Eyraud-Dubois +3

This paper introduces a new activation checkpointing method which allows to significantly decrease memory usage when training Deep Neural Networks with the back-propagation algorit…