Adam Accumulation to Reduce Memory Footprints of both Activations and Gradients for Large-scale DNN Training

Yijia Zhang, Yibo Han, Shijie Cao +5

Running out of GPU memory has become a main bottleneck for large-scale DNN training. How to reduce the memory footprint during training has received intensive research attention. W…